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I.  INTRODUCTION 


A.  PURPOSE 

The  series  of  standards  developed  and  maintained  by  the  International 
Organization  for  Standardization  (or  ISO  -  from  the  Greek  isos  meaning  “equal)i  offer  a 
collection  of  internationally  recognized  standards  as  tools  for  the  improvement  of  organ¬ 
izational  performance.  While  the  majority  of  ISO  standards  are  focused  on  a  specific 
material,  product  or  process,  the  ISO  9000  series  is  represented  as  a  “generic  standard”  in 
that  “the  same  standards  can  be  applied  to  any  organization,  large  or  small,  whatever  its 
product  -  including  whether  its  "product"  is  actually  a  service  -  in  any  sector  of  activity, 
and  whether  it  is  a  business  enterprise,  a  public  administration,  or  a  government  depart¬ 
ment.”1 2  Within  the  United  States  Department  of  Defense  acquisition  community,  the 
benefits  of  quality  management  standards  have  been  recognized  since  the  establishment 
of  the  MIL-Q-9858  quality  management  program  in  1959. 3  With  the  transition  of 
military  acquisition  to  commercial  standards  begun  by  Secretary  of  Defense  Perry  in 
1994,  the  government  expanded  the  horizons  of  defense  acquisition  by  recognizing  the 
equivalence  of  commercial  quality  management  standards,  including  ISO  9000,  for 
application  to  products  and  services  provided  by  defense  contractors. 

While  quality  management  standards  and  programs  have  long  been  applied  to 
contractors  supplying  DoD,  there  is  no  definitive  guidance  on  the  use  of  quality  manage¬ 
ment  standards  within  DoD.  The  Defense  Department  maintains  a  large  collection  of 
research  and  development  organizations  that  specialize  in  fields  and  disciplines  critical  to 
national  defense  as  well  as  organizations  with  numerous  other  functions.  Those  organi¬ 
zations  primarily  specializing  in  research  and  development  focus  on  the  specialized  needs 
of  the  military  where  there  is  insufficient  capacity  in  industry  or  the  technologies  are  too 


1  international  Organization  for  Standardization,  “About  ISO  Introduction,” 
http://www.iso.ch/iso/en/aboutiso/introduction/index.html.  Website  modified  16  February  2004,  Accessed 
April  20,  2004. 

2  Ibid. 

3  Hollis  G.  Bray,  Jr.,  “ISO  9000  in  Construction,”  Journal  of  Construction  Education,  Vol.  2,  No.  3, 
Fall  1997,  http://asciournals.org/ice/iournaEl 997/no3/Fall%201997,%20Vol.%202,%20No. 
%203.%20pp.%20 1 82- 1 92.pdf.  Accessed  April  20,  2004. 
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specialized  or  sensitive  for  application  to  the  commercial  sector.  Ultimately,  the  output 
of  these  organizations  is  translated  into  equipment,  systems  and  services  that  are  provided 
to  the  military  to  enable  the  execution  of  their  assigned  missions. 

This  case  study  will  examine  the  benefits  and  drawbacks  of  the  application  of  an 
ISO  9000  quality  management  system  within  one  such  DoD  activity  and  draw  com¬ 
parisons  with  other  applications  of  the  ISO  9001:2000  standard  within  both  industry  and 
other  government  activities.  It  will  also  address  the  efficacy  of  the  ISO  9001:2000 
standard  in  an  environment  with  diverse  customers  and  programs  with  very  low  (often 
one)  unit  production.  The  primary  focus  of  this  investigation  will  be  to  determine  if  the 
application  of  ISO  9000  to  a  DoD  laboratory  improves  efficiency,  value  and  quality  to 
the  end-user  of  the  products  delivered  as  well  as  the  taxpayer.  Comparisons  to  alternative 
quality  standards  for  Research,  Development,  Test  and  Evaluation  (RDT&E)  applications 
will  also  be  made. 

B.  RESEARCH  QUESTIONS 

1.  Primary  Research  Questions 

a.  What  is  the  Background  That  Encouraged  Institution  of  ISO 
9001  Processes  and  Certification  in  the  First  Place? 

b.  What  Has  Been  the  Outcome  of  ISO  9001?  That  is,  Has  ISO 
Had  the  Desired  Effect(S)  and  are  the  Environment/Situation/ 
Circumstances  Still  About  the  Same  That  Previously  Led  Up  to 
the  ISO  Decision? 

c.  Is  ISO  9001:2000  the  Right  Answer  for  Today’s  Acquisition 
Environment? 

d.  What  are  the  Advantages  and  Disadvantages  of  Alternative 
Quality  Management  Systems/Processes? 

2.  Secondary  Research  Questions: 

a.  Does  ISO  9001:2000  Registration  Provide  Benefits  That 
Outweigh  the  Costs  for  a  DoD  Organization,  Specifically  the 
Range  Systems  Development  Business  Unit  (RSDBU)? 

b.  Is  ISO  9001:2000  Effective  for  an  Organization,  Such  as  the 
RSDBU,  That  Has  Diverse  Programs  and  Customers? 

c.  Is  ISO  9001:2000  Effective  for  an  Organization  That  Engages  in 
Programs  with  Small  Unit  Production  Quantities? 
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d.  What  are  the  Perceived  Advantages  and  Disadvantages  of  Using 
ISO  9001  Today? 

e.  What  are  the  Similarities  and  Differences  Between  the 
Application  of  ISO  9001:2000  to  the  RSDBU  and  Other 
Organizations,  Both  Government  and  Commercial? 

f  Does  ISO  9001:2000  Certification  Increase  Competitive 

Advantage  for  the  RSDBU  as  a  Navy  Working  Capital  Fund 
(NWCF)  Activity?  Has  ISO  Certification  Increased  Funding  or 
Prevented  the  Loss  of  Funding? 

g.  Are  There  Alternative  Standards/Quality  Management  Systems 
That  are  More  Appropriate  for  DoD  Research  and  Development 
Organizations? 

h.  Is  Application  of  ISO  9001:2000  Compatible  with  an 
Organization  That  is  Highly  Structured  and  Regulated,  Such  as 
DoD? 

C.  SCOPE  OF  THIS  JOINT  APPLIED  PROJECT 

This  research  uses  the  case  of  the  Range  Systems  Development  Business  Unit. 
Analysis  will  focus  on  the  RSDBU’s  approach  to  adopting  a  quality  management  system 
and  the  effectiveness  of  the  process  and  implementation  selected. 

D.  METHODOLOGY 

In  order  to  provide  a  better  understanding  of  the  benefits  and  issues  involved  with 
implementing  ISO  9001,  this  research  paper  first  provides  an  organizational  overview  of 
RSDBU  and  the  rationale  that  went  into  selecting  a  quality  management  system.  In  order 
to  accomplish  this  we  utilized  the  following  resources: 

•  Range  Systems  Development  Business  Unit  Policy  &  Process 
Documentation 

•  Books,  periodicals,  journals,  and  electronic  resources  available  at  the 
Naval  Postgraduate  School  (NPS)  Library 

•  Internet  websites  pertaining  to  ISO  9001  and  other  Quality  Management 
Systems 

•  Interviews  with  Senior  Management  and  Team  Leaders  who  are  using  ISO 
9001 
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Then,  we  conducted  an  analysis  of  the  effectiveness  of  the  process  implemented 
by  the  RSDBU.  The  end  result  of  this  case  analysis  is  an  evaluation  on  the  effectiveness 
and  impact  of  the  ISO  9001  process  on  the  business  practices  and  program  management 
efficiencies  within  the  RSDBU. 

E.  ORGANIZATION 

Chapter  II,  provides  an  overview  of  the  specific  ISO  9001  process  adopted  by  the 
RSDBU.  This  is  followed  by  an  assessment  of  the  effects,  benefits  and  drawbacks  of 
ISO  9001:2000  certification  for  the  RSDBU. 

Chapter  III  analyzes  the  use  of  ISO  9001:2000  in  the  broader  Defense  Acquisition 
Workforce  and  explore  similarities  and  differences  in  the  application  of  ISO  9001 
between  the  RSDBU  and  other  organizations,  both  government  and  commercial. 

Chapter  IV  discusses  the  advantages  and  disadvantages  of  alternative  quality 
management  systems  and  processes  and  whether  any  of  these  alternative  systems  might 
be  more  appropriate  for  DoD  research  and  development  organizations. 

Chapter  V  summarizes  the  findings  of  this  research  in  the  form  of  conclusions 
derived  from  this  study  followed  by  recommendations  for  the  RSDBU  and  other  govern¬ 
ment  organization  for  improving  the  use  of  ISO  9001  as  an  organizational  performance 
improvement  tool.. 

F.  BENEFITS  OF  STUDY 

The  results  of  this  research  will  provide  valuable  insight  into  the  potential  applica¬ 
tion  of  ISO  9001:2000  to  government  organizations  as  a  methodology  for  continuous 
improvement  of  customer  satisfaction  and  delivered  products  and  services.  This  case 
study  will  also  examine  the  arguments  of  ISO  9001  detractors  and  assess  whether  the 
counterpoints  to  certification  have  merit  based  on  the  results  observed  in  the  period  since 
the  Naval  Undersea  Warfare  Center  (NUWC)  Division  Newport  Range  Systems 
Development  Business  Unit  has  become  certified. 
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G.  BACKGROUND  -  A  DOD  ORGANIZATION  SEEKING 

STANDARDIZATION  OF  PROCESSES 

NUWC  is  the  undersea  technology  arm  of  the  U.S.  Navy’s  largest  procurement 
organization,  the  Naval  Sea  Systems  Command,  (NAVSEA).  Headquartered  in  Newport, 
Rhode  Island,  NUWC  has  two  major  subordinate  activities:  Division  Newport  (collocated 
with  headquarters)  and  Division  Keyport  located  in  Keyport,  Washington.  In  addition  to 
its  two  main  sites,  NUWC  has  several  detachments  geographically  spread  across  North 
America  including  Andros  Island,  Bahamas;  Hawthorne,  Nevada;  Lualualei,  Hawaii;  San 
Diego,  California;  and  Nanoose,  British  Columbia,  Canada.  Established  in  1992,  NUWC 
serves  the  undersea  needs  of  both  the  surface  and  submarine  communities  by  providing 
research  and  development  of  key  undersea  technologies  to  enable  the  vision  of  providing 
“the  technical  foundation  which  enables  the  conceptualization,  research,  development, 
fielding,  modernization,  and  maintenance  of  systems  that  ensure  our  Navy's  Undersea 
Superiority. ”4  NUWC  and  its  predecessor  organizations  have  focused  on  disciplines 
related  to  undersea  warfare  for  130  years.  Division  Newport’s  roots  go  back  to  1869, 
when  the  U.S.  Navy  established  the  Naval  Torpedo  Station  in  Newport  to  experiment 
with  torpedo  technology,  explosives,  and  electricity.  Newport  predecessor  organizations 
included  the  New  London,  Connecticut  Underwater  Sound  Lab  complex  that  was  created 
during  World  War  II  to  explore  undersea  acoustic  technology.  Division  Keyport  is  the 
descendant  of  a  Torpedo  Station  that  was  established  in  Washington  in  1914. 

In  1997,  the  NUWC  Division  Newport  Executive  Director  authorized  the  merger 
of  two  departments  into  one  new  organization  to  achieve  management  efficiencies  and 
reinforce  similar  core  competencies.  Senior  management  chose  to  use  this  opportunity  to 
establish  a  new  organizational  structure  better  suited  to  current  operating  conditions  and 
patterned  after  many  successful  commercial  and  government  organizations.  The  new 
organization  was  created  using  the  Competency  Aligned  Organization/Integrated  Product 
Team  (CAO/IPT)  model,  based  on  the  principles  of  empowerment,  communications,  and 
teamwork.  The  new  department,  the  Ranges,  Engineering,  and  Analysis  Department 


4  NUWC  Intranet,  “Keeping  America’s  Navy  #1  in  the  World,”  Mission  Statement,  Updated 
June  18  2004. 
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(Code  70),  provides  full  spectrum  engineering  resources,  which  enhances  undersea 
warfare  through  the  application  of  simulation-based  development,  undersea  tracking 
range  technology  development,  engineering  analysis  and  design,  fabrication,  and  systems 
for  testing  and  training  in  representative  environments. 

A  Department  Head  and  Deputy  Department  Head  provide  the  senior  leadership 
for  Code  70.  The  Deputy  Department  Head  reports  to  the  Department  Head  while 
sharing  all  the  roles  and  responsibilities  of  the  Department  Head.  Reporting  directly  to 
the  senior  leadership  are  the  five  Competency  Leaders  and  a  varying  number  of  Team 
Leaders  whose  population  is  adjusted  based  on  workload  and  program  funding. 

A  Competency  within  this  organization  is  a  multi-disciplinary  grouping  of 
individuals  who  share  common  or  synergistic  fields  of  work  through  similar  expertise  and 
experience.  The  five  Competencies  comprising  Code  70  include  Algorithms  and 
Software  Development,  Test  Requirements  and  Conduct,  Test  Facilities  and  Operations, 
Technology  Development  and  Application,  and  Prototyping  and  Development.  The 
assignment  of  an  individual  to  a  particular  Competency  is  based  primarily  on  either  the 
field  of  work  for  which  an  individual  is  most  qualified  or  the  type  of  work  the  department 
requires  in  order  to  meet  current  needs.  Management-level  decisions  are  guided  by  the 
Leadership  Council  composed  of  the  Department  Head,  Deputy  Department  Head,  and 
the  five  Competency  Leaders.5 6  In  operation,  the  Competency  Leaders’  primary  respon- 
sibility6  is  to  provide  qualified  personnel  for  program  execution,  even  though  they,  as 
functional  leaders,  are  not  principally  involved  in  the  technical  or  programmatic  aspects 
of  program  execution.  Conversely,  Team  Leaders  are  responsible  for  the  technical  and 
programmatic  aspects  of  program  execution,  but  not  for  personnel  supervision  or 
management.  This  benefits  the  Department  and  all  of  its  personnel  by  allowing  the 
Competency  Leaders  and  the  Team  Leaders  to  concentrate  full  time  on  their  respective 
responsibilities  without  the  distraction  and  priority  conflicts  associated  with  mixing 


5  NUWCDIVNPT,  Engineering,  Test,  and  Evaluation  Department,  Code  70  Concept  of  Operations. 
Revision  5C,  January  2003,  Section  10.0  and  Appendix  B. 

6  The  Leadership  Council  is  a  body  that  meets  regularly  to  guide  Department  operations  and  to  discuss 
long-term  plans  and  policies.  This  council  is  a  key  element  of  Code  70’s  communications  toolbox  (refer  to 
footnote  #5). 
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programmatic  and  human  resources  responsibilities.  Business  Units  were  later 
established  by  forming  groups  of  Teams  focused  on  particular  aspects  of  the  business  and 
customer  bases. 

1.  The  Business  Unit  Approach 

For  many  years,  Code  70  personnel  have  been  the  Navy’s  principal  research, 
development  and  design  organization  for  the  installation  and  modernization  of  undersea 
tracking  ranges.  These  ranges  are  used  to  provide  testing  for  new  Undersea  Warfare 
(USW)  systems  and  to  support  the  U.S.  Navy  in  fulfilling  its  continuing  needs  for  USW 
proficiency  training.  Code  70  has  supported  undersea  range  customers  across  the  globe, 
including  ranges  off  the  east  coast  of  the  United  States,  in  the  Bahamas,  the  Caribbean, 
southern  California,  Hawaii,  the  Pacific  Northwest,  as  well  as  a  foreign  military  sales 
case  with  the  Australian  government.  Additionally,  Code  70  manages  and  operates  the 
Atlantic  Undersea  Test  and  Evaluation  Center  (AUTEC),  a  U.S.  Navy  range  in  the 
Bahamas,  which  is  one  of  NUWC’s  largest  programs.  The  Code  70  RSDBU  was  formed 
as  a  synergistic  collective  of  cross-functional  expertise  to  generate  work  plans  and 
execute  programs  to  maximize  both  cost  effectiveness  and  capability  delivered.  The 
RSDBU  consists  of  all  Code  70  Program  Team  Leaders  currently  managing  customer 
tasking  related  to  the  establishment,  improvement,  or  modernization  of  an  undersea 
tracking  range.  Members  of  the  RSDBU  have  experience  or  specialization  in  areas 
including  underwater  tracking,  in-water  equipment  installation,  systems  engineering, 
shore-based  infrastructure,  advanced  distributed  simulation,  and  ADP  software  and 
hardware  development.  Typical  programs  conducted  by  the  RSDBU  range  anywhere 
from  one -year  $250,000  efforts  to  multi-year  programs  exceeding  $20M  in  total  cost. 
The  membership  of  the  RSDBU  serves  as  the  peer  review  forum  for  the  sequence  of 
reviews  in  each  development  program’s  life. 

2.  Influence  from  Economic  and  Business  Factors 

This  core  competency  of  undersea  range  systems  design,  installation  and 

modernization  is  not  consistently  available  in  industry  because  of  the  limited  customer 

base  for  large  area,  highly  accurate  undersea  tracking  combined  with  declining  defense 

budgets  that  do  not  provide  a  steady  demand,  necessitating  that  this  function  be  retained 

“in-house”  within  the  government.  Despite  the  global  distribution  of  undersea  tracking 
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facilities,  the  total  number  of  facilities  is  limited  due  to  the  expense  and  complexity 
involved  in  tracking  an  object  underwater.  While  the  technologies  employed  are  similar, 
the  needs  of  the  ranges  and  their  customers  are  often  sporadic  and  divergent,  resulting  in 
systems  that  operate  from  the  same  first  principles  of  science  but  with  innumerable 
variations  of  requirements  and  implementations.  This  results  in  a  diverse  customer  base 
for  the  RSDBU,  with  differing  goals  for  technology  development  and  equally  diverse 
program  management  styles  and  requirements. 

The  primary  bifurcation  of  undersea  tracking  range  requirements  has  been  down 
the  lines  of  fleet  training  as  one  major  resource  sponsor  and  the  test  and  evaluation 
community  as  the  other.  Until  early  2004,  this  divergence  could  be  traced  all  the  way 
back  to  the  Pentagon  where  there  were  separate  resource  sponsors  for  Navy  training 
ranges  and  test  and  evaluation  ranges.  These  two  resource  sponsors  worked  through 
separate  program  management  offices  of  the  Naval  Air  Systems  Command  (NAVAIR), 
who  provided  individual  tasking  to  the  RSDBU  teams.  At  the  time  of  this  study,  the  two 
resource  sponsors  are  in  the  process  of  combining  into  a  single  Navy  Range  Office 
(NRO);  consequently,  the  program  management  structure  organization  is  currently  in 
flux.  Additional  tasking  is  received  from  other  organizations  on  a  reimbursable  basis  for 
Technical  Direction  Agent  (TDA)  support  and  other  implementations  of  undersea 
tracking  range  technology  for  missions  such  as  projectile  impact  scoring  in  support  of 
missile  testing  and  Naval  Surface  Fire  Support  (NSFS)  qualification  and  training. 

With  numerous  customers  comes  a  broad  range  of  program  management  styles 
and  reporting  requirements.  Prior  to  the  pursuit  of  ISO  9001:2000  certification,  no 
program  management  or  quality  management  process  was  required  by  the  Ranges, 
Engineering,  and  Analysis  Department  management  for  the  execution  of  programs  by  the 
RSDBU.  At  a  higher  policy  level,  there  is  little  guidance  for  government  research 
facilities  on  the  implementation  and  operation  of  a  quality  management  system  for 
products  and  services  developed  wholly  or  partially  by  government  employees. 
Customer  requirements  vary  from  the  imposition  of  a  specific  program-planning  tool  for 
budget  and  schedule  estimates  to  monthly  reporting  of  financial  expenditures  to  simply 
measuring  progress  based  on  deliverable  products.  Within  this  environment,  it  was 
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essentially  impossible  to  evaluate  the  quality  of  the  products  delivered  to  the  customer  in 
the  context  of  the  resources  consumed,  and  equally  vexing  to  derive  lessons  learned  from 
projects  that  encountered  problems  during  execution.  These  issues  extended  into  the 
estimation  of  new  work  as  well;  without  a  consistent  method  of  record-keeping,  it  was 
difficult  to  estimate  new  work  based  on  past  experience  other  than  the  corporate 
knowledge  possessed  by  senior  employees. 

An  additional  factor  necessitating  efficient  business  processes  is  that  NUWC  is  a 
Navy  Working  Capital  Fund  (NWCF)  activity  and  as  such,  operates  more  like  a  business 
than  a  traditional  government  enterprise  funded  by  a  budget  appropriation.  The  NWCF  is 
a  revolving  fund  relying  on  funded  tasking  from  external  customers  rather  than  direct 
appropriations  to  finance  its  operations.  As  such,  NUWC  must  generate  sufficient 
revenue  to  cover  the  full  costs  of  operations  and  finance  continuing  operations.  Within 
these  constraints,  NUWC  must  operate  on  a  break-even  basis  over  time  by  neither 
making  a  profit  nor  incurring  a  loss.7 

Within  the  defense  acquisition  milieu,  sponsors,  as  customers,  decide  where  to 
buy  products  and  services  based  on  performance  records  and  cost  competitiveness.  Much 
like  a  commercial  enterprise,  NUWC  gets  orders  for  work  from  our  customers,  and 
subsequently  bills  them  for  the  performance  of  that  work.  All  of  the  money  used  to 
operate  the  Warfare  Center,  maintain  infrastructure,  and  pay  employees  consists  of  funds 
received  from  customers  for  delivery  of  products  and  services.  This  contrasts  with 
institutionally,  or  mission-funded,  activities  that  are  directly  funded  by  Congressional 
appropriations. 

Because  NUWC  operates  as  a  NWCF  activity,  cost  concerns  influence  business 
decisions.  The  need  for  funded  tasking  drives  the  requirement  to  be  cost  competitive  in 
order  to  maintain  a  business  base  that  supports  the  organization  and  the  mission  it  fulfills. 


7  OSD  Comptroller  iCenter,  “A  Little  Background,”  http://www.dod.mil/comptroller/icenter/ 
dwcf/background. htm.  Accessed  April  20,  2004. 
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Therefore,  the  organizational  pricing  strategy  must  strike  a  balance  between  recovering 
costs  and  enabling  investment  and  recapitalization  required  to  maintain  the  core  mission 
capability  of  the  Center. 

Although  NUWC  operations  resemble  those  of  private  businesses,  there  are  major 
differences.  NWCF  activities  are  subject  to  controls,  including  financial,  personnel,  and 
procurement  regulations,  imposed  by  Congress,  DoD,  Department  of  the  Navy,  and 
NAVSEA  as  parent  organizations.  Furthennore,  Congress  exercises  indirect  control  by 
establishing  the  appropriation  levels  of  customer  organizations.  Customers  cannot  fund 
work  at  an  NWCF  activity  if  Congress  does  not  fund  their  programs.  Another  distinction 
is  the  NWCF  requirement  to  break  even  over  time.  The  metric  used  to  measure  organ¬ 
izational  financial  performance  toward  this  goal  is  the  Net  Operating  Result  (NOR).  This 
break-even  goal  necessitates  rigorous  cost  control  to  avoid  a  significantly  negative  NOR 
and  creates  incentives  for  the  organization  to  ensure  that  customers  are  satisfied  with 
delivered  products  and  the  associated  budget  performance. 

The  inherent  fluctuation  and  variability  in  the  business  base  for  underwater 
tracking  systems  and  components  combined  with  the  NWCF  constraints  incentivizes 
RSDBU  members  to  remain  vigilant  to  the  appearance  for  new  work  opportunities,  either 
domestically  or  abroad.  Two  Foreign  Military  Sales  (FMS)  opportunities  appeared  in 
early  2000  as  solicitations  —  one  from  the  Turkish  Navy  and  one  from  the  Royal  Navy 
for  acoustic  measurement  systems.8  However,  both  organizations  controlling  the  work 
required  potential  suppliers  to  have  an  ISO  9000  certified  quality  management  system  as 
a  precondition  to  bidding.  This  event  provided  the  initial  impetus  for  considering  certifi¬ 
cation,  since  the  lack  of  certification  had  foreclosed  a  potential  business  opportunity. 

During  this  time,  the  RSDBU  was  in  the  process  of  completing  several  major 
jobs,  including  the  $17M  task  of  replacing  all  of  the  in- water  sensors  at  the  Atlantic 
Undersea  Test  and  Evaluation  Center  (AUTEC).  The  FY2000  and  FY2002  budgets  did 
not  look  promising  from  the  perspective  of  containing  major  range  system  development 


8  Robert  J.  Reid  ,  “RE:  History  Question,”  e-mail  to  Andrew  Appleget,  29  March  2004. 
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programs  that  would  fill  the  projected  workload  gap,  so  the  RSDBU  business  develop¬ 
ment  process  began  to  emphasize  smaller  jobs  in  support  of  sustainment  of  the  core 
equity. 

3.  Pursuing  ISO  9001  Certification 

With  the  RSDBU  facing  reduced  workload  from  long-standing  traditional 
customers,  reevaluating  organizational  processes  from  the  perspective  of  establishing 
new  customers  was  necessary.  Simultaneously,  at  the  senior  management  level,  interest 
increased  in  the  documentation  of  work  processes.  Given  the  increasing  focus  on  quality 
management  in  the  commercial  sector  and  the  recently  demonstrated  requirement  from 
FMS  customers,  ISO  9000  certification  was  examined  to  detennine  if  institution  of  a 
quality  management  system  would  improve  business  processes  and  increase  the  potential 
to  capture  business  from  new  customers.  ISO  9001:2000  was  considered  in  the  context 
of  other  complementary  efforts  as  well,  including  the  Electronics  Industries  Alliance 
(EIA)  Standard  632:  Processes  for  Engineering  a  System  (EIA  632)  and  the  Capability 
Maturity  Model  Integration  (CMMI)  for  software  developed  by  the  Software  Engineering 
Institute  (SEI)  at  Carnegie  Mellon  University. 

The  initial  examination  of  alternatives  began  in  February  of  2000, 9  with  discus¬ 
sions  with  other  Division  Newport  departments  on  their  initiatives  in  achieving  ISO 
9001:2000  certification  as  well  as  scheduling  offsite  meetings  and  presentations  by 
contractors  with  expertise  in  the  implementation  of  ISO  9000  compliant  systems.  After 
several  months  of  deliberations,  the  decision  was  made  in  early  November  2000  to  form  a 
“special  projects  team”  that  would  become  ISO  9000  certified.10  The  “special  projects 
team”  would  later  become  the  RSDBU  as  the  Code  70  organization  transitioned  to  the 
grouping  of  project  teams  into  business  units.  The  team/business  unit  approach  to  ISO 
certification  was  also  highly  recommended  by  both  consultants  and  other  NUWC 
Division  Newport  organizations  that  had  pursued  and/or  achieved  ISO  certification.  This 
approach  was  perceived  to  increase  the  chances  for  success  on  the  initial  certification 
audit  by  restricting  the  scope  of  the  processes  to  be  documented  to  those  that  had  the 


9  Dale  L.  Paquette,  “Lead  Engineer  SITREP,”  e-mail  to:  #NPRI_70,  25  February  2000. 

10  Ibid. 
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greatest  commonality  based  on  the  work  perfonned  by  the  business  unit.  This  approach 
also  reduced  the  employees  required  to  gain  ISO  9000  proficiency  to  a  more  manageable 
number,  thereby,  simplifying  the  initial  training  process.  Once  initial  certification  had 
been  achieved,  additional  work  units  could  be  incorporated  based  on  the  existing  quality 
process  manual  and  lessons  learned  from  the  initial  certification  audit  and  follow  up 
compliance  audits. 

The  primary  concerns  of  pursuing  ISO  9001:2000  certification  were  the  cost 
involved  in  generating  the  quality  manual  and  guiding  procedures  documentation  as  well 
as  the  additional  overhead  that  would  be  imposed  on  programs  in  order  to  achieve  and 
maintain  compliance  with  the  ISO  quality  policy.  Additional  documentation  and 
program  review  requirements  would  effectively  increase  the  cost  to  the  sponsor  for 
program  execution.  However,  the  Ranges,  Engineering,  and  Analysis  Department  senior 
management  made  the  decision  to  commit  the  required  resources  and  proceed  with  the 
certification  effort  based  on  the  presumption  that  the  initial  investment  would  reap 
benefits  that  would  outweigh  the  costs.  In  an  e-mail  to  all  members  of  the  Department  on 
December  16,  2000,  Richard  Bonin,  Head,  NUWC  Division  Newport  Ranges, 
Engineering,  and  Analysis  Department  (Code  70)  formally  announced  the  Department’s 
commitment  to  achieving  ISO  9001:2000  certification: 

For  much  of  the  past  two  years,  we  have  examined  the  merit  of 
documenting  Code  70 ’s  various  processes.  To  date,  we  have  made  some 
progress,  especially  in  software  engineering,  but  not  nearly  enough  to  reap 
the  benefits  of  doing  so.  This  is  especially  true  in  the  field  of  program 
management  and  systems  engineering  which  are  the  mainstays  of  a 
significant  part  of  Code  70’s  business.  It  is  crucial  to  our  financial  future 
that  we  develop  and  instill  a  sense  of  discipline  into  these  areas  ensuring 
that  our  planning  and  cost  estimates  and,  therefore,  our  progress  against 
requirements,  budget,  and  schedule  are  accurate.  Perfonnance  to  plan  will 
enable  us  to  bid,  win  and  compete  for  future  projects.  We  will  avoid 
submitting  non-competitive  bids  or  underbidding  and  damaging  our 
financial  viability.  Documented  processes  will  also  enable  us  to  better 
view  and  understand  the  technical  work  being  performed  by  our  teams. 

Often,  external  project  insight  can  enable  a  team  to  take  advantage  of  the 
knowledge,  skills  and  experiences  of  others  to  identify  unseen  problems, 
insert  new  ideas,  and  to  avoid  duplicating  work  which  has  already  been 
done  elsewhere.  Most  projects  face  adversity  usually  resolved  at  some 
expense  in  time  and  money.  The  lessons  learned  in  these  experiences 
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need  to  be  documented  and  made  available  to  all  teams.  This  transfer  of 
lessons  learned  will  offer  on-going  [sic]  projects  the  opportunity  to  avoid 
[costly  mistakes]  or  rapidly  implement  the  optimum  solution  for  similar 
problems,  improving  our  efficiency.  Lastly,  we  have  already  missed  the 
opportunity  to  bid  on  two  Foreign  Military  Sales  (FMS)  jobs  because  we 
are  not  an  ISO  9001  certified  organization.  FMS  represents  a  potentially 
profitable  future  business  base  for  Code  70.  ISO  9001  certification  can 
only  be  accomplished  by  documenting  our  processes,  following  our 
processes,  and  being  able  to  prove  to  an  external  auditor  that  we  do  follow 
them.  1 1 

The  initial  Department  process  documentation  approach  was  not  solely  centered 
on  ISO  9001:2000.  The  Algorithm  and  Software  Development  Division  had  been 
pursuing  software  CMMI  process  development  and  certification  in  parallel  through 
consultations  with  SEI  representatives  under  contract  to  DoD  to  foster  increased  use  of 
the  CMMI  standard.  EIA  632  was  used  as  the  framework  for  the  development  of  the 
Design  Control  document  since  that  process  required  the  most  tailoring  to  fit  the  specific 
needs  of  the  Department.  The  original  intent  was  to  use  these  processes  as  comple¬ 
mentary  approaches  across  the  majority  of  the  Department’s  business  areas: 

In  consideration  of  the  above,  we  have  begun  several  formal  process 
documentation  efforts.  Many  of  you  are  familiar  with  our  CMM  efforts 
relating  to  software.  So  far,  we  have  four  separate  projects  requiring 
software  development  participating  in  this  effort.  You  may  also  have 
heard  of  our  EIA-632  systems  engineering  efforts.  This  document  lays  out 
a  blueprint  for  system  engineering  and  is  easily  adaptable  to  Code  70 
projects.  Finally,  there  is  ISO  9001,  we  have  formed  a  Special  Projects 
Team  and  are  diligently  working  to  get  this  team  certified  so  that  we  may 
bid  FMS  work.  Both  the  MONET  [Miniature  Optical  Node  for 
Environmental  Testing]  and  the  LSVTC  [Land-Sea  Vulnerability  Test 
Complex]  projects  are  working  with  this  team  and  will  be  the  first 
instances  of  our  doing  ISO  9001  work.  Also,  there  is  now  a  Systems 
Engineering  Functional  Team,  which  is  working  to  standardize  some  of 
the  processes  and  standards  used  in  systems  engineering.  Please  note  that 
these  various  efforts  are  fully  coordinated.  Specifically,  ISO  9001  will  be 
the  top  level  set  of  processes  with  EIA-632  and  CMM  applying  heavily  to 


11  Richard  L.  Bonin,  No  title,  e-mail  to  #NPRI_70,  16  December  2000. 
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the  technical  aspects  of  the  systems  engineering  work.  In  fact,  EIA-632 
will  invoke  CMM  when  software  engineering  and  development  are  called 
for.  12 

As  illustrated  in  this  e-mail,  the  intent  of  senior  management  was  to  develop  a 
methodology  to  standardize  processes  for  all  of  the  Department’s  business  areas  in  order 
to  retain  and  improve  the  Department’s  standing  as  the  provider  of  choice  for  the 
products  and  services  that  represent  the  mainstays  of  the  business  base.  The  Department 
Head  also  made  the  detennination  that  the  costs  of  development  of  process  documen¬ 
tation  and  the  associated  increase  in  administrative  costs  during  execution  of  a  program 
operating  under  ISO  9001 :2000  processes  were  worthwhile: 

As  the  above  work  progresses,  there  is  an  important  ramification  for  the 
rest  of  [the  undersea  range  product  side  of]  the  department.  In  order  to  be 
a  process  driven,  ISO  9001  organization  the  utilization  of  these  processes 
cannot  be  discretionary.  I  recognize  there  can  be  difficulty  in  terms  of 
time  and  expense  in  applying  these  processes  to  established  projects. 
However,  it  is  prudent  and  far  less  difficult  to  apply  them  to  newly 
beginning  projects.  I  would  like  to  inform  all  Team  Leaders,  aspiring  TLs 
and  project  team  members  that  all  new  projects  beginning  from  this  fiscal 
year  (2001)  and  on  will  be  required  to  conform  with  the  developed 
processes.  !3 

The  approach  of  senior  management  to  the  documentation  of  Department 
processes  demonstrated  a  firm  commitment  of  resources  and  effort  toward  the 
establishment  and  improvement  of  a  defined  business  model  that  would  facilitate  both 
prudent  management  decisions  as  well  as  customer  understanding  of  the  contributions  of 
the  Engineering  Test  and  Evaluation  Department  to  the  undersea  warfare  community. 
While  pursuit  of  ISO  9001:2000  certification  was  initially  motivated  by  a  desire  to 
increase  the  FMS  business  base,  added  benefits  soon  became  clear. 

The  ISO  9001:2000  certification  process  began  in  November  of  2000  with  the 
formation  of  an  internal  Department  team  to  generate  the  quality  manual  and  guiding 
procedures  documentation  and  assess  the  actions  required  to  establish  ISO  9001:2000 


12  Ibid. 

13  Ibid.,  Pt.  3. 
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compliance  amongst  the  participating  programs.14  This  team  explored  the  requirements 
for  ISO  9001:2000  certification  and  worked  to  tailor  the  implementation  of  the  ISO 
9001:2000  standard  to  the  specifics  of  the  RSDBU  business  base  as  well  as  the  policy 
requirements  of  a  DoD  organization.  During  a  March  2001  brief  to  Program  Team 
Leaders  at  a  management  offsite  meeting,  the  expected  benefits  of  adopting  an  ISO 
9001:2000  quality  system  were  defined  to  include:  the  establishment  of  Department 
credentials  to  customers,  achievement  of  greater  consistency  in  program  planning  and 
execution,  achievement  of  greater  consistency  in  technical  methods,  and  the  overall 
objective  of  consistently  making  Department  customers  successful.15 

By  June  of  2001,  the  ISO  9001:2000  documentation  process  was  nearing  com¬ 
pletion^  and  preparations  for  an  internal  compliance  audit  in  preparation  for  the  third- 
party  registrar  audit  began.  Perry  Johnson,  Inc.  was  selected  to  perform  the  registration 
audit  on  the  basis  of  corporate  familiarity  with  federal  government  organizations  seeking 
certification.  An  internal  pre-assessment  of  process  documentation  was  conducted  in 
September  of  2001  followed  by  an  internal  audit  of  RSDBU  member  program  com¬ 
pliance  and  a  mandated  management  review  in  mid-October.  The  certification  audit  took 
place  on  22-23  October  2001,  with  only  minor  non-conformances  and  one  corrective 
action  noted.  Once  these  issues  were  corrected,  the  RSDBU  was  certified  to  the  ISO 
9001:2000  standard. 


14  Robert  E.  Janiesch,  “Coordinating  Systems  Engineering,  ISO  9001  and  CMM  to  Code  70 
Programs,”  Presentation  to  Team  Leaders,  26  March  26  2001,  Slide  4. 

15  Ibid.,  Slide  3. 

l^Robert  E.  Janiesch,  “ISO  9001:2000  Rollout  for  the  Range  Systems  Development  BU,”  Presentation, 
8  June  2001,  Slide  5. 
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II.  EFFECTS,  BENEFITS  AND  DRAWBACKS  OF  ISO  9001 


A.  INTRODUCTION 

The  purpose  of  this  chapter  is  to  provide  an  overview  of  the  specific  ISO  9001 
process  adopted  by  the  Range  Systems  Development  Business  Unit  (RSDBU).  This  is 
followed  by  an  assessment  on  the  effects,  benefits  and  drawbacks  of  ISO  9001:2000 
certification  for  the  RSDBU. 

B.  OUTCOME  OF  ISO  9001  CERTIFICATION 

The  RSDBU  has  been  certified  as  an  ISO  9001:2000  compliant  organization  for 
over  two  and  a  half  years  as  of  this  writing.  As  discussed  in  Chapter  1,  the  ISO 
9001:2000  certification  process  was  undertaken  as  part  of  a  broader  process  docu¬ 
mentation  initiative  that  included  the  use  of  CMMI  for  the  software  development  group 
as  well  as  EIA  632  for  the  system  engineering  team.  Key  elements  of  these  three 
processes  were  used  to  generate  the  Design  Control  Process,  which  is  the  core  manage¬ 
ment  process  followed  by  every  program  within  the  RSDBU.  During  that  time,  the 
operating  environment  for  the  RSDBU  underwent  changes  similar  to  the  rest  of  the 
defense  acquisition  community  as  the  transformation  from  a  Cold  War  force  structure  to 
an  instrument  to  combat  global  terrorism  continues. 

The  fiscal  environment  for  the  RSDBU  has  remained  turbulent  in  the  post- 
September  11,  2001,  defense  acquisition  drive  toward  defense  transformation.  The  Chief 
of  Naval  Operations  (CNO)  has  retained  his  “Top  5  Priorities”17  as: 

1 .  Manpower 

2.  Current  Readiness 

3 .  Future  Readiness 

4.  Quality  of  Service 

5 .  Alignment 


17  Navy  Office  of  Information,  “Top  Five  Priorities,”  http://www.chinfo.navy.mil/navpalib/cno/cno- 
top5.html.  Accessed  April  20,  2004. 
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As  a  Navy  research  and  development  organization,  the  RSDBU  offers  value  to  the 
Navy  in  the  context  of  supporting  the  achievement  of  both  current  and  future  readiness. 
This  is  accomplished  by  enabling  more  effective  readiness  preparation  through  the 
development  of  undersea  tracking  systems  and  assessment  tools  for  fleet  training 
exercises  as  well  as  providing  the  facilities,  analysis  tools,  and  planning  services  to 
enable  test  and  evaluation  of  developmental  undersea  warfare  systems.  Budget  con¬ 
straints  have  intensified  in  support  of  ongoing  military  actions  associated  with  the  War  on 
Terrorism,  resulting  in  continued  pressure  to  achieve  efficiencies  in  defense  procurement 
activities.  Major  changes  are  in  process  at  the  Navy  systems  command  levels,  including 
the  realignment  of  the  NAVSEA  laboratories  to  better  pursue  the  CNO’s  Sea  Power  21 
vision. 

Within  this  strategic  context,  the  value  of  a  documented  quality  management 
system  can  provide  an  important  organizational  credential  to  demonstrate  a  commitment 
to  Navy  and  DoD  transformational  initiatives  and  the  achievement  of  mandated 
efficiencies.  The  increased  priority  on  homeland  security  and  protection  of  domestic 
infrastructure  continues  to  reinforce  the  need  to  demonstrate  the  value  added  of  all 
activities  throughout  DoD.  Utilizing  a  quality  management  system  with  inherent  process 
controls  provides  one  avenue  to  demonstrate  commitment  to  delivering  products  that  are 
optimized  to  meet  the  requirements  of  today’s  warfighter  within  the  fiscal  constraints 
imposed  by  an  evolving  threat  environment. 

At  the  operational  level,  the  requirements  of  an  NWCF  activity  and  the  business 
base  of  the  RSDBU  support  the  case  to  have  a  quality  management  system  that  efficiently 
enables  the  conduct  of  smaller  scope  defense  acquisition  programs.  There  is  no  current 
guidance  or  policy  from  the  Secretary  of  Defense  (SECDEF)  or  Secretary  of  the  Navy 
(SECNAV)  levels  on  the  internal  use  of  ISO  9000  by  NAVSEA  field  activities.18  While 
the  DoD  5000  series  of  instructions  provide  policy  guidance  on  the  implementation  of 


18  Department  of  the  Navy,  Application  of  the  International  Organization  for  Standardization  (ISO) 
9000  Series  Quality  Standards  in  NAVSEA  Programs.  NAVSEAINST  4855.33,  19  February  1997,  P  2. 
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quality  management  for  procurements  as  part  of  Major  Defense  Acquisition  Programs 
(MDAPs),  they  do  not  address  internal  government  activities  that  directly  or  indirectly 
support  MDAPs. 

The  RSDBU  business  base  is  primarily  composed  of  smaller  programs  indirectly 
supporting  MDAPs,  as  well  as  non-MDAP  tasking,  enabling  current  fleet  readiness 
through  the  provision  of  products  and  services.  In  this  context,  it  may  be  perceived  as 
having  limited  utility  to  invest  scarce  resources  in  quality  management  practices  for 
programs  that  generally  range  in  size  from  $100K  to  $20M.  However,  the  NWCF 
structure  creates  a  management  incentive  to  ensure  that  the  DoD  workforce  is  appro¬ 
priately  sized  to  meet  current  and  future  requirements  based  on  funded  tasking  from 
resource  sponsors.  The  resultant  motivation  to  minimize  overhead  and  associated 
indirect  costs  while  continuing  to  deliver  quality  products  that  support  military  customer 
requirements  mandates  that  quality  assurance  is  implemented  in  an  efficient  and  cost- 
effective  manner.  Quality  and  affordability  are  significant  driving  requirements  for  all 
organizations,  both  government  and  commercial,  seeking  to  remain  relevant  in  the 
current  DoD  fiscal  environment. 

Efficiency  motivations  also  continue  to  drive  commercial  activity  initiatives  of 
current  and  previous  political  administrations  through  assessments  of  inherently  govern¬ 
mental  activities  and  analysis  of  “core  equities.”  Core  equities  are  capabilities  that  do  not 
exist  in  private  industry  due  to  lack  of  stable  demand,  but  are  required  to  enable  the 
provision  of  affordable  military  capabilities  and  therefore  inherently  governmental.  To 
support  these  core  equities,  prudent  management  practices  dictate  the  diversification  of 
the  customer  base  to  ensure  that  these  functions  are  maintained  at  a  reasonable  cost  to  the 
taxpayer. 

An  effective  quality  management  system  supports  both  the  goals  of  an  adequate 
business  base  and  an  effectively  sized  workforce  by  providing  a  quality  management 
structure  that  can  be  tailored  to  the  smaller  size  of  the  programs  conducted  by  the 
RSDBU  and  maintained  without  onerous  indirect  cost  burdens  to  the  customer.  In  the 
international  defense  arena,  an  ISO  9001:2000  certification  serves  to  establish  the 
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imprimatur  necessary  to  enable  leveraging  of  allied  nations’  military  investments  in 
support  of  the  maintenance  of  the  RSDBU  core  equity  of  designing  and  installing 
systems  for  undersea  tracking. 

The  foregoing  evaluation  of  the  evolution  of  influences  of  the  external  environ¬ 
ment  reveals  that  the  conditions  that  initially  inspired  the  RSDBU  process  documentation 
initiatives,  including  an  ISO  9001:2000  certified  quality  management  system,  warranted 
the  pursuit  of  process  documentation  objectives.  The  ongoing  instability  of  operating  in  a 
transfonnational  environment  continues  to  validate  the  need  for  a  quality  management 
system  that  efficiently  enables  improved  deliverables  from  a  government  organization. 
Based  on  that  premise,  it  is  necessary  to  determine  if  the  process  documentation  and 
quality  management  system  implementations  have  been  beneficial  for  the  RSDBU. 

Since  the  achievement  of  ISO  9000  certification,  the  process  documentation 
initiatives  within  the  Ranges,  Engineering,  and  Analysis  Department  have  stabilized  to  an 
emphasis  on  maintaining  ISO  9001:2000  certification  as  the  primary  implementation  of  a 
quality  management  system.  EIA  632,  which  served  as  the  blueprint  for  the  system 
engineering  process,  was  used  as  the  template  for  the  development  of  the  Design  Control 
Process  for  the  RSDBU,  and  the  System  Engineering  Functional  Team  (that  drafted  the 
Design  Control  Process)  continues  to  use  that  as  a  reference  when  and  if  the  process 
requires  updates.  The  software  CMMI  certification  effort  was  deferred  after  an  initial 
qualification  audit  did  not  achieve  CMMI  Level  2  certification.  Although  several  Level  3 
and  4  Key  Perfonnance  Parameters  (KPPs)  were  being  met,  one  of  the  KPPs  required  to 
achieve  Level  2  that  was  not  being  met  was  establishment  of  an  independent  testing 
group.  Each  program  is  generally  responsible  for  planning  and  executing  the  testing  of 
all  components  being  developed  and  delivered  to  the  customer.  Establishing  and  main¬ 
taining  an  independent  testing  group  was  not  considered  a  trivial  effort.  The  conclusions 
drawn  by  the  Ranges,  Engineering,  and  Analysis  Department  from  that  audit  found  that 
the  resource  requirements  of  achieving  and  maintaining  a  CMMI  documented  process 
exceeded  the  benefits  that  would  be  obtained  for  the  business  base  of  the  Software 
Functional  Team.  Although  CMMI  Level  2  certification  is  no  longer  being  pursued  at 
this  time,  the  KPPs  that  can  be  applied  and  implemented  within  the  Software  Functional 
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Team  are  still  being  followed.  The  need  for  a  strong  process  for  dealing  with  customer 
requirements,  feedback,  and  satisfaction  convinced  the  RSDBU  to  continue  its  focus  on 
following  ISO  9001:2000  for  overall  program  management,  while  using  EIA  632  and 
CMMI  elements  to  cover  the  engineering  development  aspects  of  all  programs.19  Since 
the  Software  Functional  Team  primarily  serves  internal  customers  within  the  Code  70 
organization,  their  members  are  assigned  to  the  various  Program  Teams  of  the  RSDBU  as 
well  as  other  teams.  Since  the  RSDBU  Team  Leaders  follow  ISO  9001:2000  as  their 
driving  quality  management  requirement,  software  developers  must  be  trained  and  follow 
the  tenets  of  ISO,  which  provides  a  level  of  process  confonnity.  While  this  process 
conformity  is  not  specifically  tailored  to  the  software  development  environment,  it 
provides  enough  guidance  to  ensure  that  the  needs  of  individual  programs  are  met  and  the 
software  products  delivered  fulfill  the  requirements  of  the  overall  delivered  product. 

The  Code  70  quality  management  system  has  continued  to  mature  through  the 
continuous  improvement  tenets  of  the  ISO  9001:2000  standard  and  expand  to  other  areas 
of  department  business.  The  Dodge  Pond  Acoustic  Testing  Facility,  another  portion  of 
the  Code  70  enterprise  that  conducts  underwater  acoustic  transducer  testing  and  calibra¬ 
tion,  implemented  a  process  documentation  initiative  and  was  certified  to  the  ISO 
9001:2000  standard  in  June  2004.  Within  the  RSDBU,  periodic  internal  surveillance 
audits  required  to  maintain  ISO  9001:2000  certification  discovered  process  anomalies 
and  areas  where  corrective  actions  were  not  entirely  successful.  The  most  recent  internal 
audit  conducted  in  March  2004,  discovered  three  instances  of  nonconformance  to 
process,  five  observations  and  one  major  nonconformance  resulting  in  the  reopening  of  a 
corrective  action  report  (CAR).20  Resolution  of  the  CAR  required  the  development  and 
implementation  of  an  additional  work  instruction  to  fonnalize  the  process  for  getting 
internal  customer  concurrence  on  requirements  for  development  programs  delivering 


19  Boris  Mutafelija  and  Harvey  Stromberg,  “ISO  9001:2000  —  CMMI  vl.l  Mappings,”  Carnegie 
Mellon  Software  Engineering  Institute,  July  2003,  http://www.sei.cmii.edu/cmmi/adoption/iso- 
mapping.pdf,  Accessed  April  20,  2004. 

20  NUWCDIVNPT,  Code  70,  “ISO  9001:2000  Management  Review,  Range  Systems  Development 
Business  Unit  Review  Meeting,”  Presentation  briefing,  16  March  2004,  P.  3-6. 
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products  to  the  AUTEC  range.  These  results  indicate  that  Code  70  management  remains 
committed  to  the  retention  of  ISO  9001:2000  certification  by  maintaining  the  continuous 
improvement  process. 

From  a  process  maintenance  perspective,  the  direct  costs  of  the  Code  70  ISO 
9001:2000  system  have  remained  a  worthwhile  investment  for  Code  70.  Currently,  the 
ISO  9000  management  representative  role  is  filled  as  an  additional  duty  by  the  Deputy 
Competency  Leader  who  is  also  the  leader  of  the  RSDBU,  requiring  no  additional 
financial  investment  as  the  Deputy  position  is  overhead  funded.  The  internal  audit  team 
is  composed  of  one  government  employee  working  part-time  on  ISO  tasking  and 
contractor  consultation  for  ISO  9000  process  and  documentation  expertise.  These  labor 
costs,  combined  with  the  fee  for  the  annual  third  party  certification  audit,  total 
approximately  $50,000  for  FY04 .21 

During  FY04,  the  RSDBU  is  executing  $19.5M  of  customer  funding  including 
$7.6M  of  Code  70  internal  labor, 22  so  the  traceable  direct  cost  of  maintaining  certification 
represents  0.3%  of  the  total  RSDBU  budget..  This  comparatively  miniscule  investment 
has  proven  sufficient  to  sustain  RSDBU  processes  at  a  level  necessary  to  maintain 
certification  as  well  as  managing  the  administration  of  the  program  review  process.  It  is 
not  possible  to  detennine  the  indirect  costs  to  individual  programs  for  maintaining  ISO 
compliance  without  adding  unreasonable  burden  on  Team  Leaders  and  team  members. 
An  attempt  to  definitively  quantify  the  additional  costs  imposed  by  the  RSDBU  ISO 
process  would  be  counterproductive  from  the  perspective  of  differentiating  efforts 
directly  attributed  to  ISO  9001:2000  compliance  as  opposed  to  non-ISO  required 
program  management  activities. 

While  the  indirect  compliance  costs  to  individual  programs  cannot  easily  be  deter¬ 
mined,  it  has  been  the  authors’  experience  that  there  is  little  additional  administrative  and 
financial  burden  added  by  RSDBU  quality  management  system  requirements.  Most  of 
the  documentation  required  by  the  RSDBU  quality  management  system  is  consistent  with 


21  Doris  Carvalho,  phonecon,  15April  2004. 

22  Code  70  financial  data  and  AUTEC  I&M  program  status  briefs  derived  from  multiple  internal 
sources. 
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sound  program  management  practice,  the  additional  ISO-specific  requirements  such  as 
cover  sheets  and  tracking  tags,  create  negligible  impacts  on  employees’  time  and  pro¬ 
ductivity.  Productivity  gains  can  also  be  achieved  through  better  program  organization 
driven  by  the  ISO  process. 

The  most  significant  program  impact  is  found  in  preparing  for  the  conduct  of 
reviews  mandated  by  the  Program  Review  Process  work  instruction.  Since  RSDBU 
sponsors  require  varying  levels  of  progress  and  status  reporting  detail  from  Team 
Leaders,  fulfilling  the  RSDBU  ISO  program  review  requirements  can  consume  additional 
time  in  collecting  and  formatting  ISO-required  information  at  the  level  of  detail  specified 
in  the  instruction.  Program  reviews  generally  occur  one  to  two  times  per  year  and  are 
event-driven  by  the  progress  of  the  program;  a  total  of  six  reviews  (excluding  the  initial 
task  review)  are  required  by  the  RSDBU  process. 

Based  on  the  authors’  experience,  no  more  than  20  -  40  total  team  work-hours  are 
usually  required  to  prepare  for  each  review  depending  on  the  size  and  complexity  of  the 
program.  This  level  of  effort  is  not  a  significant  expense  for  RSDBU  programs  and  has 
been  streamlined,  in  comparison  to  previous  program  review  methods,  by  the  RSDBU 
ISO  9001:2000  quality  system.  The  use  of  a  common  program  review  format  provides  a 
documented  baseline  to  which  Team  Leaders  can  plan,  and  thereby  budget,  when 
programs  are  proposed.  A  common  process  facilitates  the  mapping  of  data  elements  from 
program  planning  and  execution  tools  to  the  program  review  requirements  with  nominal 
additional  effort  required  to  maintain  ISO  9001  compliance.  ISO  9001  specific  require¬ 
ments  are  generally  limited  to  fulfilling  documentation  requirements,  such  as  verifying 
record  cover  sheets  and  ensuring  document  revisions  are  current. 

From  an  external  perspective,  it  is  challenging  to  quantitatively  assess  specific 
benefits  that  have  been  achieved  in  customer  satisfaction  or  changes  in  the  quality  of 
delivered  products  as  a  result  of  achieving  ISO  9001:2000  certification.  As  part  of  the 
ISO  9001:2000  processes  adopted  by  the  RSDBU,  as  illustrated  in  Figure  1,  the  measure¬ 
ment  of  customer  satisfaction  is  a  required  input  to  the  establishment  and  monitoring  of 
business  objectives.  The  most  cost  effective  way  to  assess  customer  satisfaction  is 

through  the  use  of  surveys  focused  on  specific  factors  of  interest,  as  opposed  to  the 
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alternative  of  conducting  individual  interviews,  which  can  be  costly  in  terms  of  schedule, 
travel,  and  lost  productivity. 

NUWC  Division  Newport  has  instituted  a  Division-wide  customer  survey  process 
in  support  of  the  unit  self-assessment  requirements  of  the  NAVSEA  Inspector  General. 23 
Customers,  in  the  fonn  of  principal  stakeholders  (including  fleet  commands  and  type 
commanders)  and  resource  sponsors,  are  annually  provided  with  a  customer  survey  to 
determine  their  satisfaction  with  the  products  and  services  delivered  by  Division  Newport 
organizational  entities.  Sponsors  to  be  surveyed  are  derived  from  assignment  numbers 
allocated  by  the  financial  system  and  correspond  to  those  people  responsible  for 
providing  funding  documents  while  stakeholders  are  identified  by  senior  management. 


Figure  1.  Code  70  Process  Interaction  Diagram 


23  NAVSEA,  “NAVSEA  Command  Performance  Inspection  (CPI)  Team  6  Improvements,” 
21  September  1999. 
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Historically,  the  survey  responses  from  Code  70  customers  have  averaged  a  30  - 
50%  rate  of  return  and  involve  less  than  20  individual  responses,  so  it  is  difficult  to 
derive  statically  significant  data.  For  example,  the  most  current  data  available  for  FY02 
showed  an  increase  in  customer  rated  quality  of  delivered  products  of  8%  over  FY01,  as 
opposed  to  a  3.6%  decrease  in  customer  rated  quality  from  FY01  to  FY00.24  However, 
since  the  Code  70  survey  data  also  includes  customers  of  non-RSDBU  members  it  is 
impossible  to  quantitatively  assess  the  impact  of  ISO  9001:2000  certification  on  customer 
satisfaction  based  on  these  survey  results.  A  lack  of  “before  and  after”  pairings  of  survey 
responses  from  a  sufficient  number  of  either  ISO-controlled  or  non-ISO-controlled 
programs  prohibit  this  quantitative  assessment.  This  issue  was  noted  as  an  ongoing 
observation  during  the  past  two  Code  70  ISO  management  reviews.  As  a  result,  alterna¬ 
tive  customer  survey  methodologies  are  being  considered  that  are  tailored  to  Code  70- 
specific  business  objectives  management  and  which  more  directly  support  the  ISO 
process. 

The  primary  metrics  used  by  most  customers  to  evaluate  RSDBU  perfonnance  are 
cost,  schedule  and  performance.  Some  programs  are  also  evaluated  on  budget  execution 
since  that  is  a  metric  used  by  comptrollers  of  the  Navy,  other  services,  and  DoD  to 
evaluate  program  managers  that  provide  tasking  to  the  RSDBU.  While  the  RSDBU  ISO 
9001:2000  system  is  not  aimed  at  facilitating  budget  execution,  it  does  address  cost, 
schedule  and  perfonnance  as  components  of  product  quality  and  earned  value.  Also,  the 
preponderance  of  RSDBU  customers  are  government  activities  that  do  not  require  a 
quality  management  system  as  a  precondition  for  tasking  and  do  not  operate  internally 
under  a  quality  management  system. 

Therefore,  the  RSDBU  ISO  9001:2000  quality  management  system  seems 
unlikely  to  influence  customers’  decisions  on  providing  tasking  to  the  RSDBU  on  the 
basis  of  their  organizational  culture  or  program  management  requirements.  It  has  been 
the  authors’  experience  that  RSDBU  operation  under  an  ISO  9001:2000  quality  manage- 


24  Code  70  customer  survey  data  FY02  summary,  compiled  and  retained  within  Code  70 
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ment  system  has  not  been  used  as  a  marketing  tool  with  most  sponsors  and  at  most  has 
been  only  peripherally  mentioned  during  the  course  of  sponsor  program  reviews. 

From  this  analysis,  the  RSDBU  ISO  9001:2000  quality  system  is  unlikely  to  have 
a  significant  impact  on  customer  perception  of  value  added,  so  it  is  necessary  to  consider 
if  there  is  an  indirect  benefit  created  by  facilitating  more  effective  execution  of  sponsor 
tasking  through  the  improvement  of  internal  RSDBU  processes.  At  a  top  level,  this 
objective  can  be  found  in  the  Code  70  Quality  Policy: 

Our  goal  is  to  consistently  provide  systems  that  satisfy  our  customers  by 
meeting  or  exceeding  their  quality  requirements  and  expectations 
including,  but  not  limited  to,  system  performance,  cost  and  schedule.  This 
will  be  accomplished  by  maintaining  a  constant  commitment  to  process 
evaluation,  innovation  and  continuous  improvement.25 

This  policy  provides  the  foundation  for  the  Code  70  quality  management  system 
and  summarizes  the  intended  value  added  of  the  system. 

Another  method  to  assess  the  overall  effectiveness  of  the  Code  70  system  is  by 
examining  the  progress  achieved  toward  the  original  senior  management  expectations  of 
certification  benefits  as  laid  out  in  the  Code  70  policy  issued  in  December  of  2001.  The 
e-mail  promulgating  the  Code  70  process  documentation  policy  laid  out  four  major 
expected  benefits:2^ 

•  Instill  discipline  in  planning  and  cost  estimates 

•  Better  understanding  of  technical  work  being  performed  across  the 
organization 

•  Ability  to  bid  on  FMS  work  requiring  ISO  9000  certification 

•  Documentation  of  lessons  learned 

The  overall  perceived  advantage  of  a  process  documentation  approach  was  based 
on  using  ISO  9000,  CMMI  and  EIA-632  processes  to  accomplish  improved  organiza¬ 
tional  performance  of  program  management  and  system  engineering  functions  and 

25  NUWCDVINPT,  Code  70  Departmental  Business  Unit  and  Cost  Center  Quality  Processes  Manual. 
(Level  I  Document).  3  May  2004,  P.  24. 

26  Richard  L.  Bonin,  No  title,  e-mail  to  #NPRI_70,  16  December  2000. 
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improving  the  product  realization  activities  (such  as  software  and  digital  signal 
processing  system  development)  that  are  significant  components  of  most  RSDBU  deliv¬ 
erable  products.  The  following  evaluates  each  of  the  expected  benefits  based  on  30 
months  of  experience  with  the  process  to  date. 

1.  Instill  Discipline  in  Planning  and  Cost  Estimates 

The  overall  RSDBU  process,  as  illustrated  in  Figure  1  and  detailed  in  the  Program 
Review  Process  work  instruction,  mandates  a  Requirements  Review  prior  to  commence¬ 
ment  of  new  work.27  This  review  is  conducted  upon  receipt  of  a  funded  tasking  to  ensure 
that  the  project  was  adequately  scoped  within  the  bounds  of  available  technology,  work¬ 
force  resources,  and  has  a  realistic  budget  and  schedule.  However,  since  this  review 
occurs  after  the  receipt  of  a  funded  tasking,  the  proposal  that  was  forwarded  in  order  to 
receive  the  tasking  has  already  been  generated  and  accepted  by  the  customer,  thereby 
creating  an  agreement  that  has  established  customer  expectations  for  cost,  schedule  and 
performance.  Therefore,  the  Requirements  Review,  as  the  first  component  of  the  Design 
Control  Process,  does  not  fulfill  the  goal  of  ensuring  that  the  proposal  provided  was 
adequately  scoped  in  terms  of  deliverables,  schedule,  and  cost,  since  the  RSDBU  had 
effectively  accepted  the  tasking  by  the  time  the  Requirements  Review  was  conducted. 

The  solution  to  this  issue  was  the  inclusion  of  a  separate  review,  called  a  Task 
Review,  which  is  required  “when  the  proposal  under  consideration  has  a  value  of  at  least 
$200,000  or  is  proposing  at  least  one  half  man  year  of  labor  for  Code  70. ”28  This  meth¬ 
odology  was  adopted  to  provide  greater  consistency  and  rigor  in  proposal  review  and  to 
ensure  that  varying  aspects  of  a  project  were  appropriately  scoped  prior  to  a  proposal 
being  released  externally  to  potential  sponsors. 

The  Task  Review  can  be  conducted  in  one  of  three  of  ways:  a  formal  presentation 
to  available  business  unit  members,  electronic  distribution  to  business  unit  members  for 
review  and  comment,  or  expedited  review  by  Code  70  senior  management  for  cases 
where  an  informal  proposal  or  rough  order  of  magnitude  estimate  is  required  on  short 

27  NUWCDIVNPT,  Ranges,  Engineering  and  Analysis  Department  (Code  70)  Range  Systems 
Development  Business  Unit,  Design  Control,  Doc.  No.  RSD-QMP-73-01,  Rev.  3,  24  February  2004,  P.  17. 

28  NUWCDIVNPT,  Code  70,  Task  Review  Work  Instruction,  Doc.  No.  RSD-QMP-72-02,  Rev.  4, 

17  November  2004,  P.  2. 
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notice.29  The  multiple  modalities  of  the  task  review  process  provide  flexibility  to  ensure 
an  appropriate  level  of  scrutiny  is  applied  based  on  the  scope  and  size  of  the  proposal, 
potential  labor  resource  impacts,  and  the  time  available  for  proposal  generation.  The 
electronic  distribution  alternative  also  helps  mitigate  the  constraints  of  a  geographically 
distributed  organization  and  the  frequent  travel  schedules  of  RSDBU  members. 

The  task  review  process  is  designed  to  include  representation  from  all  of  the 
RSDBU  disciplines,  such  as  in-water  installation,  shore  systems  electronics,  digital  signal 
processing,  and  ADP  software,  to  review  a  proposal  for  adequacy  of  cost,  schedule,  and 
resource  estimates.  This  approach  takes  into  account  the  experiences  and  lessons  learned 
over  a  broad  spectrum  of  disciplines.  Since  the  RSDBU  is  composed  of  a  number  of 
specialists  in  varying  disciplines,  it  is  challenging  for  any  individual  member  to  develop  a 
proposal  with  a  broad  scope  of  work,  based  only  on  their  individual  experience. 

Fifteen  task  reviews  have  been  conducted  since  the  inception  of  the  task  review 
process  in  September  2001.30  Of  those,  seven  have  been  formal  presentations,  six  have 
been  conducted  via  e-mail  and  two  were  completed  by  senior  management  review. 
Attendance  at  formal  reviews  averaged  20-25  employees,  including  RSDBU  members, 
team  members  and  management.  The  comments  generated  as  a  result  of  these  reviews 
range  from  notations  of  grammatical  errors,  to  missing  documentation  or  information 
required  by  the  task  review  process,  to  perceived  technical  shortfalls  or  implementation 
risks  of  the  proposed  scope  of  work. 

Since  the  proposal  is  reviewed  by  employees  with  a  broad  collection  of  skills  and 
experience,  portions  of  the  proposal  that  may  be  unclear  to  people  who  are  not  specialists 
in  the  primary  discipline  of  the  proposal  will  be  revealed  before  the  proposal  is 
submitted.  A  proposal  may  be  perfectly  clear  to  the  employee  who  generated  the 
proposal,  but  by  having  the  proposal  reviewed  by  employees  with  a  different  skill  mix 
and  technical  perspective  provides  a  “sanity-check”  to  ensure  that  the  intent  of  the 
proposal  is  clearly  communicated  to  someone  without  a  specialized  technical  background 


29  ibid.,  P.  3. 

30  Information  resides  on  Code  70  ISO  Folder  located  on  global  drive  (\\Npri70fs\global\ISO\Review 
MinutesYTask  Reviews). 
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in  the  subject  of  the  proposal,  as  is  often  the  case  with  resource  sponsor  personnel.  The 
use  of  a  broad-based  internal  peer  and  management  review  also  helps  motivate 
employees  generating  proposals  to  “sharpen  the  pencil”  and  ensure  that  planning  and 
estimation  are  accomplished  with  sufficient  rigor  before  a  proposal  is  released. 

The  task  review  process  has  been  updated  three  times  since  its  initial  release31  and 
a  fourth  revision  is  in  process  to  expand  the  scope  of  the  process  to  include  proposals 
submitted  for  Major  Range  Test  Facility  Base  (MRTFB)  funding,  which  are  currently 
exempted.  While  some  proposals  have  been  released  without  the  benefit  of  a  task  review, 
conformance  has  generally  been  increasing  as  the  new  process  gains  wider  comprehen¬ 
sion  by  RSDBU  members  and  is  reinforced  by  management.  Overall,  this  process  can  be 
judged  as  meeting  initial  expectations  for  improving  the  rigor  of  planning  and  cost 
estimates. 

2.  Better  Understanding  of  Technical  Work  Being  Performed  Across  the 
Organization 

The  evolution  of  Code  70  as  the  Ranges,  Engineering,  and  Analysis  Department 
has  brought  together  a  diverse  collection  of  technical  specialties  and  facilities  within  a 
common  management  structure.  However,  because  of  this  diversity  it  is  challenging,  if 
not  impossible,  for  employees  or  senior  management  to  maintain  an  in-depth  currency  in 
the  science  and  technology  that  is  being  employed  across  the  entire  organization.  This 
situation  created  the  initial  motivation  for  the  formation  of  business  units  as  a  fonn  of 
“communities  of  practice”  for  areas  of  tasking  with  technical  and/or  customer  com¬ 
monality.  The  business  unit  construct  also  proved  convenient  when  it  was  necessary  to 
identify  a  collection  of  people  and  programs  for  the  purpose  of  seeking  ISO  9001:2000 
certification  by  identifying  the  RSDBU  as  the  organization  to  be  certified  and  developing 
processes  tailored  to  that  specific  business  area. 

The  program  review  process  mandates  a  comprehensive  overview  of  the  current 
program  status  including  a  summary  of  program  documentation  content  and  external 
review  status,  a  summary  of  the  tasks  accomplished  for  the  phase(s)  being  reviewed, 


31  Task  Review  Work  Instruction.  Doc.  No.  RSD-QMP-72-02,  Rev.  4,  17  November  2004,  P.  1. 
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status  of  Environmental  Review  Board  approvals,  significant  program  issues,  significant 
technical  issues,  an  assessment  of  personnel  resources  available  for  executing  the 

program,  and  current  risk  assessment  and  mitigation  plans. 32  These  components  are 
mandatory  for  all  programs  and  provide  senior  management  with  a  convenient  and 
standardized  oversight  mechanism. 

The  RSDBU  is  composed  of  a  wide  array  of  programs,  including  the  imple¬ 
mentation  of  advanced  modeling  and  simulation  systems  at  open-air  tracking  ranges, 
ADP  software  development,  acoustic  telemetry  system  design,  in-water  sensor  fabrica¬ 
tion  and  installation,  digital  signal  processing  development,  and  electronics  design.  The 
function  of  the  RSDBU  is  to  ensure  that  these  and  other  elements  are  combined  to  deliver 
a  cohesive  and  optimized  level  of  capability  to  the  Navy  in  a  cost-effective  and  timely 
manner  through  leveraging  of  technologies  developed  across  the  spectrum  of  individual 
tasks  received  from  sponsors  with  often  disparate  requirements. 

Against  this  backdrop,  the  RSDBU  program  review  process  provides  a  common 
template  as  a  progress  review  mechanism  and  facilitates  increased  understanding  of 
technical  work  being  performed  across  the  organization.  The  structure  of  the  program 
review  process  requires  that  Team  Leaders  present  the  technical  approach  and  evolving 
design  status  over  the  course  of  the  program  life  cycle  until  the  system  or  capability  is 
delivered.  Also,  since  attendance  of  the  all  of  the  available  business  unit  members  is 
required  at  Design  Control  Process  reviews,  these  reviews  also  serve  as  ersatz  technical 
forums  to  provide  a  nominal  level  of  education  to  other  business  unit  members  on  the 
current  status  of  various  technologies  that  are  critical  to  the  RSDBU  business  base.  The 
review  process  also  serves  to  help  educate  new  employees  in  areas  outside  their  technical 
specialization.  An  added  benefit  of  the  review  process  is  in  the  designation  of  ISO 
program  reviews  as  fulfilling  continuous  learning  objectives  for  the  Defense  Acquisition 
Workforce  Improvement  Act  (DAWIA)  program  required  for  all  Code  70  employees. 


32  NUWC  Division  Newport,  Code  70,  Program  Review  Process  Work  Instruction,  Doc.  No.  RSD-WI- 
DES-01,  Rev.  3,  20  October  2003,  P.  1. 
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3.  Ability  to  Bid  on  FMS  Work  Requiring  ISO  9000  Certification 

Although  competing  for  FMS  opportunities  was  the  initial  impetus  to  pursue 
certification,  it  has  turned  out  to  be  one  of  the  areas  with  the  least  return  on  investment. 
Since  achieving  ISO  9001:2000  certification,  the  RSDBU  has  not  gained  any  FMS  work 
for  which  ISO  9000  certification  was  a  prerequisite.  However,  Code  70  currently  has  six 
active  data  exchange  agreements  with  Australia,  Sweden,  Norway,  France,  the  Republic 
of  Korea  and  Japan33  that  have  the  potential  to  lead  to  FMS  business  opportunities.  FMS 
cases  tend  to  take  a  significant  amount  of  time  to  develop  due  to  the  inherent  com¬ 
plexities  of  international  transfer  of  defense  technology  and  the  associated  regulatory  and 
policy  coordination.  Also,  given  the  changes  in  the  global  defense  posture  as  a  result  of 
the  War  on  Terrorism,  many  countries  are  scaling  back  their  investments  in  undersea 
tracking  technology  as  their  priorities  have  shifted  to  homeland  defense  and  anti¬ 
terrorism  initiatives. 

4.  Documentation  of  Lessons  Learned 

The  Program  Review  Process  also  serves  to  establish  a  common  framework  for 
documenting  lessons  learned  from  a  program  execution  perspective.  Having  a  standard¬ 
ized  documentation  format  serves  to  ensure  that  all  programs  provide  the  same  types  and 
amounts  of  data  that  can  be  analyzed  and  compared  for  future  reference. 

Before  the  establishment  of  the  RSDBU  and  ISO  9001:2000  certification, 
program  documentation  fonnat  and  possession  was  the  sole  purview  of  the  Team  Leader 
running  the  program,  usually  driven  by  sponsor  requirements.  Often,  when  employees 
transferred  to  other  jobs  or  retired,  their  records  were  inherited  by  remaining  employees 
or  put  into  storage.  For  those  receiving  such  an  inheritance,  filing  systems  tended  to  be 
personality  specific  and  it  was  often  difficult  and  time  consuming  to  attempt  to 
reconstruct  program  history  from  the  remaining  documentation.  Within  the  RSDBU  ISO 
process,  program  documentation  requirements  are  specific  and  consistent  for  all 
programs,  thus  simplifying  the  reconstruction  of  past  experience  and  corporate 
knowledge.  Individuals  specializing  in  program  documentation  are  becoming  more 


33  NUWC  Division  Newport,  Code  70,  “Training,  Test  and  Evaluation  Department,  Code  70, 
Department  Brief.”  Overview  presentation,  10  June  1999. 
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proficient  through  the  documentation  standardization  and  process  control.  Also,  the  shift 
to  electronic  file  storage  helps  to  facilitate  access  through  the  filing  of  program  data 
records  in  a  common  location. 

As  with  others,  this  facet  of  process  control  will  take  time  to  gain  footing  and 
provide  benefit  as  programs  transition  to  the  new  documentation  paradigm  and  corporate 
history  migrates  to  a  standardized  fonnat.  While  no  documentation  effort  can  efficiently 
capture  all  corporate  knowledge  resident  with  technical  experts,  a  common  format  will 
increase  the  productivity  of  both  current  and  new  employees  as  they  seek  to  benefit  from 
previous  experience. 

The  previous  examples  illustrate  the  results  achieved  by  RSDBU  ISO  9001:2000 
certification  in  the  context  of  the  benefits  expected  by  senior  management  at  the  time  the 
decision  was  made  to  pursue  certification.  However,  there  have  also  been  other  effects  of 
certification  beyond  the  scope  of  those  initially  anticipated. 

Consistency  of  process  has  proven  beneficial  by  providing  documented  and 
uniform  expectations  of  team  leaders  and  team  members  not  only  in  program  planning 
but  also  for  program  execution.  Since  the  ISO  9001:2000  mandate  is  management  driven 
externally  to  a  program,  potential  conflict  amongst  team  leaders  and  members  is  reduced 
or  eliminated  with  respect  to  program  reporting  requirements.  For  functional  teams  that 
work  across  multiple  programs,  such  as  software  development,  there  is  also  consistency 
from  program  to  program  on  reporting  requirements  as  well  as  the  conduct  of  peer 
reviews  in  support  of  the  program  review  process.  For  Team  Leaders  managing  more 
than  one  program,  the  RSDBU  ISO  9001:2000  process  provides  similar  benefits  in  the 
imposition  of  consistent  status  reporting  and  review  development  products. 

The  nature  of  the  RSDBU  ISO  9001:2000  implementation  also  lends  itself  well  to 
program  execution  due  to  the  inherent  tailoring  that  occurs  in  any  instantiation  of  an  ISO 
9000  process.  Thus,  work  instructions  and  procedures  are  generated  only  for  those 
elements  that  require  process  documentation  above  and  beyond  existing  policy  or  regu¬ 
latory  guidance.  For  example,  since  purchasing  in  a  DoD  organization  is  controlled  by 
Federal  Acquisition  Regulations  (FAR)  and  Defense  Federal  Acquisition  Regulation 
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Supplement  (DFARS)  directives,  there  is  no  need  to  generate  separate  work  instructions 
for  the  conduct  of  purchasing  since  the  FAR  and  DFARS  are  acceptable  to  ISO  9000 
registrars  as  documented  procedures. 

The  consistency  of  the  RSDBU  ISO  9001:2000  process  also  provides  benefits 
during  personnel  changes  for  team  members  and  leaders.  Since  every  program  is  docu¬ 
mented  to  the  same  standard  and  undergoes  the  same  set  of  internal  reviews,  the  process 
of  replacing  personnel  due  to  transfers  or  retirements  is  simplified  for  the  person 
assuming  new  responsibilities.  This  provides  efficiencies  to  the  program  sponsor  by 
reducing  the  learning  curve  as  well  as  removing  any  concerns  the  sponsor  may  have 
about  the  process  that  the  new  team  leader  will  be  using  to  manage  their  program. 

Finally,  in  the  age  of  “rightsizing,”  an  ISO  9001:2000  certified  quality  manage¬ 
ment  system  helps  to  fulfill  the  requirement  of  commercial  activities  studies  for  the  Most 
Efficient  Organization  (MEO)  to  have  a  Quality  Assurance  Surveillance  Plan  (QASP). 
“The  QASP  describes  methods  of  inspection,  perfonnance  metrics,  required  reports,  and 
the  needed  resources,  including  estimated  work  hours,  to  verify  quality”  and  is  “key  to 
the  successful  inspection  of  contract,  ISSA  [Interservice  Support  Agreements],  or  in- 
house  performance.”34  A  government  organization  that  has  previously  implemented  an 
ISO  9001  compliant  quality  management  system  can  save  the  time  and  expense  of 
meeting  the  QASP  requirement  in  an  expedient  fashion  within  the  schedule  of  a 
commercial  activities  study;  thereby,  increasing  the  chances  of  being  selected  as  the 
MEO. 

In  summary,  the  conclusion  can  be  drawn  that  the  implementation  of  an  ISO 
9001:2000  quality  management  system  has  provided  benefits  for  the  RSDBU  as  an 
organization.  The  evolution  of  the  external  strategic  environment  of  the  RSDBU  has 
increased  the  need  to  be  able  to  demonstrate  organizational  value  added  to  the  Navy,  and 
a  documented  quality  management  system  contributes  to  the  achievement  of  that  goal. 


34  Chief  of  Naval  Operations,  “Navy  Commercial  Activities  (CA)  Program,”  OPNAVINST  4860. 7C, 
7  June  1999,  P.  1-24. 
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To  date,  the  most  significant  drawback  is  that  the  establishment  of  a  certified 
RSDBU  quality  management  system  cannot  be  shown  to  have  achieved  a  direct  financial 
return  since  it  is  not  provable  that  ISO  9001:2000  certification  directly  led  to  new  work 
or  the  retention  of  existing  work.  However,  it  has  been  shown  that  the  improvement  in 
business  unit  processes  provides  value  to  customers  in  terms  of  business  practices  that 
create  consistent  program  execution  requirements,  heightened  management  visibility  of 
program  performance  and  the  associated  employee  motivation  to  ensure  customer 
expectations  are  achieved. 

The  next  step  is  to  assess  the  effectiveness  of  the  RSDBU  ISO  900 1 :2000  quality 
management  system  in  the  execution  of  programs. 

C.  EFFECTIVENESS  OF  ISO  9001:2000  FOR  THE  RSDBU 

As  previously  stated,  the  RSDBU  operates  across  a  diverse  set  of  product  reali¬ 
zation  disciplines  when  compared  to  many  other  organizations  that  have  been  certified  to 
the  ISO  9001:2000  standard.  Since  the  ISO  9001:2000  standard  evolved  from  a  product 
manufacturing  background,  it  is  necessary  to  examine  the  effectiveness  of  the  standard 
when  used  in  an  organization  that  is  primarily  focused  on  research  and  development  with 
commensurately  increased  levels  of  risk  and  uncertainty  in  comparison  to  a  strictly 
production  oriented  organization.  The  necessity  for  the  RSDBU  to  cultivate  a  diverse  set 
of  customers  and  deliverable  products  to  maintain  the  core  capability  of  the  workforce 
could  lead  to  the  conclusion  that  a  common  quality  management  system  is  a  suboptimal 
solution. 

In  this  context,  it  may  seem  counterintuitive  to  apply  a  quality  management 
standard  that  emphasizes  confonnity  to  an  organization  whose  outputs  are  markedly  non- 
conforming  in  tenns  of  a  specific  product  or  narrowly-focused  product  line.  Since  the 
ISO  9001:2000  process  emphasizes  conformity,  inspection  and  corrective  action,  an 
assessment  of  effectiveness  may  be  found  in  the  examination  of  programs  that  have 
experienced  problems  in  achieving  cost,  schedule  or  performance  milestones. 

Since  the  achievement  of  the  RSDBU  ISO  9001:2000  certification,  only  one 
program  has  been  terminated  by  a  sponsor  because  of  unplanned  cost  growth.  The 
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AUTEC  Minefield  and  Shallow  Water  Range  (AMSWR)  program  was  originally  estab¬ 
lished  to  install  a  cable  from  an  existing  in-water  tracking  hydrophone  array  to  a  shore 
facility  for  Commander,  US  Atlantic  Fleet  (COMLANTFLT)  as  the  customer.  Cabling 

this  array  to  shore  would  alleviate  the  need  and  expense  of  mobilizing  and  operating  a 
range  support  vessel  to  conduct  underwater  tracking  exercises  at  a  remote  site  in  the 
Berry  Islands,  Bahamas. 

In  executing  to  the  RSDBU  ISO  9001:2000  process,  the  AMSWR  program 
conducted  a  combined  requirements  and  technical  approach  review  in  August  of  2002. 
The  review  minutes  note  that  four  technical  issues  were  identified  and  some  additional 
ISO-specific  formatting  and  documentation  deficiencies  were  also  noted.35  The  risks 
identified  in  the  initial  phase  included  ongoing  negotiations  with  the  owners  of  the  shore 
facility  where  the  shore  equipment  was  proposed  for  location  and  issues  of  cost- 
effectiveness  of  constructing  a  permanent  building  as  opposed  to  using  a  portable 
equipment  van.36  These  elements  were  identified  as  “medium  risk.” 

When  the  Preliminary  Design  Review  was  conducted  in  July  of  2003,  the  medium 
risk  elements  of  the  program  had  expanded  to  include  underwater  cable  installation 
contract  cost  overruns  or  delays  and  additional  cost  associated  with  construction  of  a 
permanent  building. 37  The  post-review  comments  were  primarily  focused  on  accurate 
documentation  of  program  status  within  the  RSDBU  ISO  format.  Review  comments  also 
noted  that  the  program  was  being  executed  with  “the  most  mature  program  management 
processes  [seen  to  date]  within  Code  70.  The  bi-weekly  updates  of  risk  assessments  and 
monthly  status  updates  to  the  program  plan  are  great  examples.”38 

Some  of  the  established  program  risks  materialized  when  the  in-water  cable 
installation  cost  estimates  from  commercial  cable-laying  vessel  operators  increased 


35  NUWCDIVNPT,  “Code  70  Range  Systems  Development  Business  Unit  Technical 
Approach/Program  Plan  Review  Meeting  Minutes,”  12  August  2002. 

36  Robert  J.  Reid,  “AUTEC  Shallow  Water  Range  &  Minefield  (ASWRM)  -  ISO  9001  Requirements  & 
Technical  Approach  Review,”  Presentation  briefing,  12  August  2002,  P.12-13. 

37  NUWCDIVNPT,  AMSWR  Team,  Code  70.  “AUTEC  Minefield  &  Shallow  Water  Range  (Berry 
Island  Range),”  Presentation,  17  July  2003,  P.  42. 

38  Debra  J.  Greenhalgh,  “AMSWR  ISO  PDR  Review  Updates  (Comments),”  email  to  Doris  M. 
Carvalho,  5  August  2003. 
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significantly  over  the  original  estimates,  as  well  as  increases  in  shore  facility  construction 
and  leasing  costs  discovered  through  continued  negotiations  with  owners  of  the  shore 
facility  and  the  government  of  the  Commonwealth  of  the  Bahamas.  The  Bahamian 
government  subsequently  added  additional  environmental  requirements  as  a  result  of 
evolving  national  policy  initiatives.  Also,  during  the  component  development  and 
fabrication  phase,  the  Navy  found  other  methods  of  conducting  operations  in  that  area 
that  reduced  the  need  for  in-water  tracking,  effectively  lowering  the  priority  for  the 
capability  to  be  provided  by  the  AMSWR  program. 

The  increased  costs  of  system  installation,  combined  with  the  potential  schedule 
delays  associated  with  the  uncertainty  of  achieving  a  final  agreement  with  the  public  and 
private  parties  involved,  ultimately  led  to  the  decision  to  defer  the  program  indefinitely 
and  reallocate  program  funding  to  other  priorities.  The  AMSWR  program  had  complied 
with  the  RSDBU  ISO  9001:2000  process,  but  it  is  unlikely  that  the  totality  of  the 
resultant  complications  that  led  to  the  program’s  termination  could  have  been  predicted 
regardless  of  the  program  planning  or  quality  management  system  in  use.  The  AMSWR 
system  design  was  of  sufficient  technical  maturity  to  be  considered  executable,  but  the 
program  tennination  decision  was  based  on  multiple  contributing  factors  outside  the 
technical  realm.  While  cost  growth  was  a  significant  root  cause  contributor  to  the  final 
decision,  there  were  also  additional  legal,  environmental  and  international  negotiation 
complexities  that  served  to  induce  unacceptable  schedule  risk  in  addition  to  the  cost 
growth  issues. 

This  example  illustrates  a  situation  where  the  RSDBU  ISO  9001:2000  process 
was  unlikely  to  provide  a  method  to  identify  the  major  causal  factors  that  led  to  program 
tennination  that  were  outside  the  product  development  and  delivery  process.  The  ISO 
9000  focus  on  process  conformity  tends  to  emphasize  product  realization  activities  in  the 
context  of  achieving  consistent  and  repeatable  processes,  enabling  requirements  trace- 
ability  and  improving  customer  satisfaction.  ISO  9000  is  not  ideally  suited  for  the 
identification  of  external  environmental  or  political  factors,  other  than  requiring  a  risk 
management  methodology,  to  address  issues  that  have  the  potential  to  derail  overall 
program  execution.  Since  risk  management  methodologies  are  solely  at  the  discretion  of 
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the  organization  operating  within  an  ISO  9000  quality  management  system,  the  results 
will  be  proportional  to  the  quality  of  the  specific  risk  implementation  used.  In  the 

AMSWR  case,  the  risk  management  was  considered  to  be  comprehensive  and  proactive, 
but  the  program  termination  decision  was  based  on  a  combination  of  individual  factors  as 
opposed  to  one  specific  risk  element. 

There  is  no  quality  management  system  in  existence  that  will  ensure  the  success 
of  any  undertaking,  particularly  in  complex  development  environments.  In  the  RSDBU 
ISO  9001:2000  system,  the  consistency  and  repeatability  aspects  are  primarily  aimed  at 
program  management  level  tasks  as  opposed  to  product  realization  tasks.  This  structure 
is  intended  to  facilitate  the  accomplishment  of  customer  objectives  and  maximize 
customer  satisfaction.  The  AMSWR  program  team  fully  complied  with  the  ISO  process 
requirements  and  the  customer  was  not  dissatisfied  with  the  performance  of  the  RSDBU 
team  assigned. 

In  instances  where  a  customer  is  dissatisfied,  the  ISO  process  provides  a  defined 
methodology  to  resolve  customer  issues  and  develop  a  course  of  corrective  action  that  is 
documented,  traceable  and  maintains  an  appropriate  level  of  visibility  at  the  management 
level.  This  portion  of  the  RSDBU  quality  management  system  was  tested  with  the 
Aberdeen  Data  Acquisition  System  (ADAS)  program. 

The  Digital  Signal  Processing  (DSP)  team  of  the  RSDBU  accepted  tasking  to 
develop  a  high-speed  digital  data  acquisition  system  for  the  U.S.  Army  Aberdeen  Test 
Center  (ATC)  to  satisfy  a  requirement  to  support  ballistic  and  survivability  and  lethality 
measurements.  Due  to  the  highly  specialized  nature  of  the  testing  performed  at  ATC,  no 
commercially-available  data  acquisition  systems  were  available  that  would  fulfill  their 
requirements.  The  DSP  team  undertook  the  task  of  developing  and  integrating  a  system 
based  on  the  similarities  to  acoustic  digital  signal  processing  architectures  used  in  under¬ 
water  tracking  range  systems  and  also  to  support  the  maintenance  of  the  government’s 
core  equity  in  specialized  data  acquisition  systems. 

Since  this  program  was  initiated  prior  to  the  implementation  of  the  Task  Review 
process,  no  initial  review  of  the  proposal  submitted  by  the  DSP  team  was  conducted. 
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During  the  course  of  execution  of  the  program,  numerous  shortfalls  in  the  program 
documentation  were  consistently  noted  in  the  design  review  minutes.  In  comparison  to 
the  AMSWR  program,  the  ADAS  team  did  not  employ  a  mature  risk  management 
approach;  in  fact,  risk  management  was  not  presented  at  any  of  the  program  reviews 
although  required  by  the  RSDBU  ISO  process. 

As  the  program  progressed,  schedule  slips,  perfonnance  shortfalls  and  cost  over¬ 
runs  occurred  due  to  a  number  of  reasons,  including  personnel  availability,  COTS 
(commercial-off-the-shelf)  technology  limitations  and  inadequate  requirements  specifi¬ 
cation.  Ultimately,  this  resulted  in  a  complaint  from  the  ATC  customer  that  “product 
performance,  specifically  timing  and  software  stability,  is  not  meeting  specifications  after 
significant  program  delay. ”39 

In  the  “pre-ISO  9000”  operations  of  the  RSDBU,  there  was  no  defined  method¬ 
ology  for  the  handling  of  customer  complaints;  this  situation  would  have  been  handled 
informally  by  the  Team  Leader  and  Code  70  management  depending  on  the  level  at 
which  the  complaint  was  lodged.  A  program  review  may  or  may  not  have  been 
conducted,  at  the  management  team’s  discretion,  to  assess  the  situation.  The  Naval 
Undersea  Warfare  Center,  Division  Newport  (NUWCDIVNPT)  uses  a  customer  survey 
process  that  allows  customers  to  express  their  satisfaction  with  the  products  and  services 
delivered.  These  surveys  are  conducted  only  on  an  annual  solicitation  basis,  which 
generally  serves  to  identify  a  customer’s  level  of  satisfaction  after-the-fact,  thereby 
significantly  reducing  opportunities  for  any  necessary  corrective  action  during  program 
execution. 

However,  under  the  ISO  9001:2000  quality  system  of  the  RSDBU,  a  defined 
process  exists  for  handling  and  resolving  customer  complaints  as  laid  out  in  the  Customer 
Feedback  and  Complaints  procedure.  The  specific  direction  provided  states  that:  “[t]he 
Team  Leader  for  the  project  or  program  evaluates  and  classifies  each  complaint,  which 
are  [sic]  then  forwarded  to  the  MR  [Code  70  ISO  Management  Representative]. 


39  NUWCDIVNPT,  Code  70,  “ISO  9001:2000  Management  Review,  Range  Systems  Development 
Business  Unit  Review  Meeting,”  Presentation  briefing,  26  March  2004. 
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Complaints  are  classified  into  categories  to  allow  for  better  tracking  of  trends  and  evalu¬ 
ating  improvement  in  specific  areas.”40  Based  on  the  Team  Leader’s  evaluation,  an 
appropriate  response  to  the  complaint  is  determined.  In  cases  where  the  complaint  is 
actionable,  courses  of  action  are  prescribed  by  the  Corrective  and  Preventive  Action 
procedure.  As  the  procedure  name  implies,  either  a  corrective  action  or  preventative 
action  is  initiated. 

The  initial  course  of  action  prescribed  by  the  Corrective  and  Preventive  Action 
procedure  is  to  determine  the  root  cause  of  the  customer’s  complaint  and  develop  an 
appropriate  remediation  strategy.  The  root  cause  of  problems  can  be  traced  to  various 
levels  of  the  organization,  therefore  requiring  varying  approaches  to  rectification.  For 
example,  defective  delivered  products  may  be  the  result  of  a  faulty  or  insufficient  pro¬ 
duction  specification  or  a  flaw  in  the  production  system  itself.  In  many  instances, 
production  work  is  subcontracted  out  and  remediation  must  go  through  the  government 
contracting  process  as  dictated  by  the  FAR  and  DFARS  regulations.  Defective  product 
handling  is  further  specified  by  another  procedure,  Control  of  Nonconforming  Product, 
which  specifies  the  methodology  for  handling  and  processing  nonconforming  products. 

The  ADAS  customer  complaint  was  based  on  perfonnance  of  prototype  system 
components  that  had  not  yet  entered  production.  The  RSDBU  procedure  specifies  that: 
“[t]he  Team  Leader  for  the  project  or  program  reviews  the  customer  complaint  to  deter¬ 
mine  whether  it  calls  for  an  internal  investigation  and  whether  it  should  be  followed  up 
with  a  fonnal  Corrective  Action  Request  (CAR).”41  In  this  case,  a  CAR  was  opened  and 
the  corrective  action  required  was  determined  to  be  the  assignment  of  an  expert  in  digital 
signal  processing  and  field  programmable  gate  arrays  to  modify  the  system  design;  along 
with  the  execution  of  a  strategy  to  verily  that  hardware  viability  for  the  required 
application  was  communicated  to  the  customer.42 


40  NUWCDIVNPT,  Code  70,  Customer  Feedback  and  Complaints,  Doc.  No.  DBU-QMP-72-01, 
Rev.  0,23  March  2004,  P.3. 

41  Ibid. 

42  “ISO  9001:2000  Management  Review,  Range  Systems  Development  Business  Unit  Review 
Meeting,”  Presentation  briefing,  26  March  2004,  P.  13. 
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Once  this  remediation  action  was  undertaken,  the  ADAS  program  rectified  the 
technical  shortfalls  noted  by  the  customer  and  delivered  a  prototype  system  for  eval¬ 
uation.  The  program  was  deemed  complete  by  the  customer  and  closed  out  by  the 
RSDBU  after  the  preliminary  design  review. 

As  illustrated  in  the  ADAS  example,  the  RSDBU  quality  management  system 
provided  a  defined  path  toward  the  identification,  correction  and  documentation  of 
customer  satisfaction  issues.  At  first  glance,  the  process  is  somewhat  reminiscent  of  a 
government  regulatory  approach,  invoking  multiple  procedures  with  numerous  paths  to 
fulfill  the  satisfaction  requirements  of  each  succeeding  procedure;  thereby,  creating  the 
potential  for  confusion,  time-consuming  lost  productivity  or  outright  disregard. 
However,  the  primary  difference  in  the  approach  of  the  RSDBU  quality  management 
system  is  the  relative  simplicity  of  the  procedures;  most  procedures  are  six  pages  or  less 
and  have  concisely  worded  instructions  on  actions  to  be  taken  by  the  user  as  well  as 
references  to  the  RSDBU  Management  Representative  for  clarification. 

Because  of  the  emphasis  of  the  RSDBU  ISO  9001:2000  system  on  program 
management  activities,  it  is  highly  dependent  on  the  personnel  executing  the  program  to 
ensure  success;  ISO  9001:2000  provides  only  the  template.  The  program  review  process 
provides  the  mechanism  to  facilitate  progress  and  status  visibility  at  the  management  and 
peer  level.  However,  since  the  program  review  process  is  primarily  event  driven,  it  is 
possible  for  problems  to  go  undetected  for  significant  periods  of  time.  The  process 
requires  an  in-process  review  if  over  one  year  has  passed  since  the  last  review.  In  the 
authors’  opinion,  this  is  a  shortfall  of  the  RSDBU  ISO  9001:2000  implementation,  since 
programs  execute  on  an  annual  budget  cycle.  Therefore,  it  is  possible  for  a  troubled 
program  to  go  through  an  entire  year  without  management  oversight  or  peer  review, 
allowing  problems  to  grow  in  terms  of  both  schedule  and  budget  resources  required  for 
correction.  A  shift  to  a  six-month  review  cycle,  with  abbreviated  in-process  review 
requirements,  might  better  serve  the  intended  purpose  of  the  program  review  process. 

The  previous  two  examples  serve  to  demonstrate  the  operation  of  the  RSDBU 

ISO  9001:2000  quality  management  system  from  the  perspective  of  programs  that  have 

encountered  problems  in  achieving  cost,  schedule  or  performance  milestones.  They  also 
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serve  to  demonstrate  that  the  RSDBU  ISO  9001:2000  quality  management  system  is 
flexible  enough  to  accommodate  widely  divergent  program  management  styles  with 
broadly  varying  degrees  of  focus  on  specific  program  details,  such  as  risk  management. 
In  parallel,  13  other  programs  operating  within  the  RSDBU  ISO  9001:2000  process  have 
completed  deliveries  or  continue  to  execute  within  cost,  schedule  and  performance 
parameters.  These  programs  range  in  size  and  duration  from  $325K  one-year  efforts  to 
$10.1M  five-year  development  programs  and  include  a  broad  array  of  deliverables  that 
range  from  prototype  system  designs  to  complex  system  development,  integration, 
delivery,  and  site  acceptance. 

The  AMSWR  and  ADAS  examples  also  help  to  illustrate  the  diversity  of  the 
RSDBU  business  base.  In  addition  to  these  two  programs,  other  sponsors  include  the 
Naval  Air  Systems  Command,  the  Office  of  Naval  Research,  the  Office  of  the  Secretary 
of  Defense,  Director  of  Operational  Test  and  Evaluation  (DOT&E),  the  NAVSEA  Test 
and  Evaluation  Office  and  the  Army  Program  Manager  for  Instrumentation,  Targets  and 
Threat  Simulators  (PM-ITTS).  As  would  be  expected,  program  management  styles  and 
requirements  vary  significantly  across  this  group. 

Therefore,  the  resultant  ISO  9001:2000  implementation  was  developed  by 
necessity  to  target  the  lowest  common  denominator  of  program  management  require¬ 
ments,  focusing  on  requirements  traceability,  product  realization  and  customer  satisfac¬ 
tion  as  the  minimum  set  of  requirements  for  all  programs.  To  have  gone  into  a  greater 
level  of  specific  process  implementation  would  only  have  served  to  make  the  process 
unwieldy  and  less  applicable  to  some  components  of  the  business  base  and  would  likely 
have  eroded  the  support  and  endorsement  of  the  RSDBU  users.  Ultimately,  program 
success  cannot  be  attributed  to  the  RSDBU  ISO  9001:2000  process  any  more  than 
program  failure;  the  quality  management  system  provides  only  a  tool  for  management  to 
evaluate  progress  and  status.  Consequently,  the  effects  of  the  RSDBU  quality  manage¬ 
ment  system  on  overall  organizational  perfonnance  will  be  determined  by  the 
effectiveness  with  which  the  quality  system  is  used  by  senior  management  to  monitor  and 
influence  program  development  and  execution. 
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D.  ISO  9001:2000  AND  SMALL  UNIT  PRODUCTION  ACTIVITIES 

With  few  exceptions,  the  RSDBU  has  executed  and  continues  to  execute 
programs  that  have  a  unit  production  quantity  of  one  delivered  end  product  or  system. 
This  is  primarily  driven  by  two  factors:  the  size  and  complexity  of  the  underwater  range 
systems  and  the  small  number  of  underwater  ranges  both  domestically  and  globally. 

Currently,  the  U.S.  Navy  operates  four  major  underwater  tracking  ranges  in  or 
near  U.S.  waters,  and  there  are  several  other  small-scale,  special-purpose  underwater 
measurement  installations  also  in  existence.  The  in-water  sensor  system  components  of 
underwater  ranges  are  generally  designed  for  a  20-year  life  expectancy  due  to  the  signifi¬ 
cant  expense  associated  with  the  specialized  equipment  and  ships  required  for  in-water 
equipment  installation.  These  costs  serve  as  a  deterrent  to  repair  of  failed  in-water 
system  components  since  it  quickly  becomes  more  cost-effective  to  simply  replace  a 
failed  component  as  opposed  to  the  costs  and  time  required  for  retrieval,  diagnosis  and 
correction. 

Automated  Data  Processing  (ADP)  components  of  underwater  range  installations 
have  shorter  life  spans  due  to  the  rapid  pace  of  advancement  in  the  commercial  ADP 
equipment  sector.  However,  due  to  the  continually  tight  fiscal  constraints  of  the  DoD 
operating  environment,  range  ADP  equipment  is  often  not  replaced  until  it  is  past  obso¬ 
lescence  and  no  longer  economically  feasible  to  repair  or  maintain.  For  example,  the 
major  ADP  system  components  at  the  AUTEC  range  were  last  replaced  in  1995  and  the 
ongoing  replacement  program  is  a  five-year,  $6M  effort  due  to  the  requirement  to  funda¬ 
mentally  re-architect  the  whole  system  because  of  the  leap-forward  in  technology 
implementation.  This  program  will  complete  in  2006,  making  it  1 1  years  (or  viewed 
another  way:  at  least  three  generations  of  computing  technology)  between  major  ADP 
technology  upgrades  for  the  AUTEC  range. 

Given  the  limited  frequency  with  which  these  types  of  programs  occur,  it  is 
difficult  to  define  a  consistent  program  execution  process  that  will  remain  relevant 
because  of  the  major  changes  in  industry  and  technology  that  occur  during  the  periods 
between  similar  major  programs.  The  net  effect  is  that  no  two  programs  will  ever  be 

executed  the  same  way.  As  a  result,  the  RSDBU  ISO  9001:2000  quality  management 
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system  must  be,  and  has  been,  structured  to  accommodate  the  advances  of  technology 
and  evolving  requirements  of  programs  with  small  unit  production  quantities. 

The  two  key  characteristics  enabling  ISO  9001:2000  to  be  used  effectively  in  the 
low  unit  production  environment  of  the  RSDBU  can  best  be  described  as  tailoring  and 
generality.  These  characteristics  support  the  approach  to  ISO  9001:2000  implementation 
that  “[t]he  complexity  and  interaction  of  processes  and  the  competence  of  the  personnel 
determine  the  scope  of  the  documented  system.  ”43 

Tailoring  allows  procedures  and  work  instructions  to  be  developed  to  the  appro¬ 
priate  level  of  detail  based  on  the  functions  of  the  organization  and  the  complexity  of  the 
tasks  undertaken.  For  example,  a  principal  emphasis  for  the  RSDBU  quality  system  is 
the  standardization  of  general  program  management  processes  and  system  engineering 
practices  across  the  business  unit.  Thus,  the  majority  of  the  effort  in  procedure  develop¬ 
ment  was  devoted  to  the  Design  Control  Process,  resulting  in  an  8  3 -page  set  of 
procedures  as  compared  to  the  average  of  six  pages  each  for  most  other  RSDBU 
procedures. 

The  Design  Control  Process  was  developed  based  on  a  combination  of  current 
program  management  and  system  engineering  practices  that  have  been  effective  in 
executing  RSDBU  programs  and  augmented  by  the  EIA  632  standard.  The  Design 
Control  Process  does  not  focus  on  the  production  quantity  of  the  program  and  is  intended 
to  ensure  that  system  designs  maximize  the  achievement  of  customer  performance 
objectives  within  cost  and  schedule  constraints.  In  this  context,  the  portion  of  the  Design 
Control  Process  dealing  with  system  production  and  implementation  does  not 
specifically  address  production  issues  that  would  normally  be  associated  with  large- 
volume  production  programs,  such  as  specialized  tooling  or  manufacturing  spaces, 
because  those  are  not  significant  considerations  for  programs  executed  by  the  RSDBU. 
The  guidance  of  section  17.2.5  simply  states,  “buy  it,  build  it,  integrate  it,”44  thereby 
leaving  the  specifics  of  production  methodology  to  the  program  team. 


43  NUWCDIVNPT,  Code  70,  Summary  of  Range  Systems  Development  Business  Unit  (RSDBU), 
Department  Business  Unit  And  Cost  Ccnter(s)  (DBU)  Level  II  Document,  2  June  2004,  P.  1. 

44  Design  Control,”  Doc.  No.  RSD-QMP-73-01,  Rev.  3,  24  February  2004,  P.  74. 
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The  generality  aspect  of  ISO  9001:2000  provides  the  flexibility  to  target 
processes  at  the  appropriate  level  of  the  organization  based  on  the  overall  competence 
level  of  the  workforce  involved  in  program  execution.  Members  of  the  RSDBU,  with 
very  few  exceptions,  possess  a  minimum  of  an  undergraduate  degree  in  a  technical 
discipline,  primarily  engineering  and  computer  science.  They  have  also  been  trained  and 
certified  to  Defense  Acquisition  Workforce  Improvement  Act  (DAWIA)  standards  to 
meet  the  requirements  of  NUWC  Division  Newport  policy  objectives.  All  of  the  RSDBU 
Team  Leaders  have  been  certified  to  DAWIA  Level  III  for  Systems  Planning,  Research, 
Development  and  Engineering.  For  these  reasons,  it  is  safe  to  assume  a  baseline  level  of 
employee  competence  in  systems  development  and  realization  activities  that  alleviate  the 
necessity  to  define  RSDBU  work  instructions  and  procedures  to  overly  detailed  levels  of 
specificity. 

In  large-volume  production  activities  for  which  ISO  9000  was  initially  conceived, 
it  is  highly  probable  that  a  more  diverse  skill  mix  of  employees  would  be  involved,  such 
as  would  be  found  in  a  manufacturing  setting  in  industry.  If  the  ISO  9001  standard 
mandated  a  total  level  of  process  documentation  structured  for  the  minimum  level  of 
competence  in  an  organization,  work  instruction  and  procedure  generation  would  quickly 
become  burdensome,  inefficient  and  costly.  However,  since  the  standards  and  inter¬ 
pretations  of  ISO  9001  registrars  compensate  for  the  level  of  competence  in  an 
organization,  documentation  content  and  effort  can  be  appropriately  scoped. 

To  date,  the  fact  that  RSDBU  programs  produce  limited  quantities  of  delivered 
products  has  not  been  a  significant  factor  in  the  overall  effectiveness  of  the  ISO 
9001 :2000  implementation.  The  system  production  aspect  of  the  Design  Control  Process 
has  been  targeted  toward  the  final  integration,  testing,  installation  and  site  acceptance 
tasks  that  are  necessary  for  the  types  of  systems  produced.  The  business  base  of  the 
RSDBU  makes  it  difficult  to  compare  one  program  to  another,  but  the  structure  of  the 
RSDBU  quality  management  system  facilitates  the  identification  of  those  program  phases 
that  present  the  greatest  challenges  or  offer  the  greatest  success.  The  ISO  9001:2000 
approach  allows  the  RSDBU,  as  consumers  of  the  standard,  the  flexibility  to  bypass  those 
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elements  associated  with  large  volume  production  and  tailor  processes  to  maximize  value 
to  the  members  and  customers  of  the  RSDBU,  regardless  of  the  production  quantity. 

As  discussed  herein,  the  RSDBU  implementation  of  ISO  9001:2000  has  proven 
beneficial  to  the  organization  by  providing  a  measurement  and  management  tool  that  has 
been  tailored  to  the  overall  focus  of  the  RSDBU  business  base.  By  definition,  the  system 
cannot  be  optimized  for  every  program  and  customer  due  to  the  diversity  of  the  business 
base  that  drives  the  necessary  generality  of  the  quality  management  system.  Nonetheless, 
the  initial  investment  in  a  quality  management  system  can  often  be  daunting  for  an 
organization  depending  on  the  scope  of  the  processes  involved,  especially  since  it  is 
difficult  to  specifically  quantify  the  actual  business  performance  gains  achievable  from 
the  system.  This  aspect  will  be  discussed  in  more  detail  in  Chapter  III. 
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III.  ISO  9001  IN  THE  DEFENSE  ACQUISITION  WORKFORCE 


A.  INTRODUCTION 

The  purpose  of  this  chapter  is  to  analyze  the  use  of  ISO  9001:2000  in  the  broader 
Defense  Acquisition  Workforce  and  explore  similarities  and  differences  in  the  application 
of  ISO  9001  between  the  RSDBU  and  other  organizations,  both  government  and 
commercial. 

B.  ADVANTAGES  AND  DISADVANTAGES  OF  USING  ISO  9001  TODAY 

The  quality  management  community  has  given  significant  attention  to  the  pros 
and  cons  of  ISO  9000  certification  as  the  number  of  certified  organizations  has  grown 
significantly  over  recent  years.  Numerous  studies  of  the  effects  of  ISO  9000  certification 
have  been  conducted  to  determine  if  there  is  a  direct  and  traceable  link  from  certification 
to  business  benefits.  These  studies  primarily  seek  to  compare  the  actual  results  of  ISO 
9000  certification  in  contrast  to  the  expected  benefits  cited  by  organizations  when  the 
decision  was  made  to  pursue  certification. 

Proponents  of  ISO  certification  extol  the  benefits  that  increased  quality  has  on 
business  performance  based  on  the  assertion  that  an  ISO  9000  certified  quality  manage¬ 
ment  system  directly  contributes  to  increased  quality  of  the  end  product  or  service 
delivered.  Critics  of  the  ISO  9000  approach  dispute  the  correlation  between  ISO  certifi¬ 
cation  and  product  quality  as  being  significantly  more  complex  than  simply  having  a 
certified  quality  management  system.  They  contend  that  it  is  possible  to  have  an 
adequately  documented  set  of  processes  that  produce  well-documented  bad  results,  such 
as  the  oft-cited  example  of  a  very  efficient  producer  of  concrete  lifejackets.45 

Similarly,  the  pursuit  of  certification  because  it  is  required  by  customers  (some¬ 
times  referred  to  as  “marketplace  coercion”)  has  been  the  subject  of  much  debate  and 
criticism  by  some  quality  management  analysts.  They  generally  contend  that  the  benefits 
of  certification  are  inflated  by  many  ISO  9000  registrars  and  consultants  who  earn  their 

45  DiverDiver,  e-government@large,  “e-Government@large,  2004  Is  The  Year  It  All  Goes 
Mainstream,”  24  September  2002,  http://www.diverdiver.com/2002  09  25  diverdiver  archive.html. 
Accessed  April  20,  2004. 
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living  from  the  ISO  process  and  that  there  is  a  wide  degree  of  variability  in  quality 
amongst  ISO  9000  certified  companies. 

Organizations  that  claim  benefits  of  certification  include  the  International 
Standards  Organization  (ISO),  the  American  Registrar  Accreditation  Board,  the  National 
Quality  Assurance  certification  body  of  the  UK  and  the  British  Standards  Institute. 
While  the  ISO  makes  no  explicit  claims  past  the  fact  that  “organizations  that  implement 
ISO  9000  voluntarily  are  doing  so  because  they  expect  these  standards  to  help  them  do 
things  better  and  to  provide  real  benefit,”46  the  certification  and  registrar  communities  go 
a  step  further.  Claims  that  certified  quality  management  systems  can  “[ljower  costs 
through  reduced  wastage  and  quality  improvement”47  and  provide  “increased  market 
share  through  perceived  higher  quality  and/or  improved  market  opportunities”48  are 
frequent  in  literature  and  promotional  materials  developed  by  these  organizations. 

An  example  of  this  situation  can  be  found  in  material  from  Vinca,  LLC,  who  are 
providers  of  ISO  9000  certification  resource  materials.  They  cite  a  case  study  published 
in  the  Dallas  Business  Journal  crediting  ISO  9000  certification  with  increased  pro¬ 
ductivity,  reduced  scrap  and  waste,  record  sales  levels  and  better  incoming  raw  material 
screening.49  What  they  fail  to  clarify  is  that  this  case  study  is  based  on  the  experience  of 
only  one  company  with  two-dozen  employees.  This  sort  of  generalization  serves  to 
reinforce  the  contention  that  the  benefits  of  ISO  certification  may  be  artificially  inflated 
by  some  organizations  that  have  a  vested  interest  in  promoting  ISO  9000. 

The  British  Standards  Institute  has  been  somewhat  more  rigorous  with  case 
studies  of  companies  in  the  service,  manufacturing  or  raw  materials  processing  sectors.50 
A  review  of  18  of  these  studies  revealed  that  17  companies  claimed  increased  customer 
satisfaction,  13  improved  internal  operations,  12  saw  improved  product  quality,  nine 

46  Gavin  P.  M.  Dick,  “Certification  Benefits,  Reality  or  Myth?,”  ISO  9000,  The  TQM  Magazine, 

Vol.  12,  Issue  6,  2000,  ISSN  0954-478X,  P.  2,  http://konstanza.emeraldinsight.com/ 
vl=3310045/cl=42/nw=l/rpsv/tqm.htm,  Accessed  April  20,  2004. 

47  Ibid. 

48  Ibid. 

49  Cynthia  Weber,  Quest  Analytical,  Inc.,  “Benefits  of  ISO  9000...,  ”  Presentation,  15  March  2004, 
http://www.the9000store.com/benefits  of  iso.ppt.  Accessed  April  23,  2004. 

50  BSI  Management  Systems,  “Case  Study  List,”  http://emea.bsi-global.com/Ouality/CaseStudies/ 
CaseStudyList.xalter,  Accessed  April  23,  2004. 
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reduced  costs,  and  eight  increased  profits.51  The  businesses  surveyed  included  financial 
services,  utilities/energy,  property  management,  marketing,  transportation,  construction 
and  telecommunications  companies  with  a  range  from  100  to  71,000  employees. 
Interestingly,  amongst  the  above  claims  of  improved  business  results,  at  least  half  of  the 
companies  surveyed  cited  customer  pressure  or  explicit  customer  requirements  for  ISO 
9000  as  one  of  the  motivations  to  pursue  certification,  lending  credence  to  those  that 
claim  “marketplace  coercion”  as  a  major  driver  in  increased  ISO  9000  registrations. 

The  vast  majority  of  study  material  generated  by  commercial  organizations 
focuses  on  industry  applications  of  ISO  9000,  since  that  is  currently  the  major  market  for 
consultants  and  registrars.  As  can  be  seen  by  the  examples  cited  above,  it  is  possible  to 
interpret  study  data  in  a  manner  consistent  with  the  goals  of  the  organization  using  the 
data.  The  Vinca,  LLC,  use  of  a  single  case  study  is  presented  in  a  far  more  generalized 
fashion  without  noting  the  limited  sample  upon  which  the  conclusions  are  based  is  one 
example.  Similarly,  the  BSI  cases  concentrate  on  businesses  that  generally  have  well- 
defined  procedures  due  to  regulatory  and/or  business  requirements  and  are  primarily 
focused  on  a  single  business  area.  However,  their  results  do  not  reflect  a  strict  attribution 
of  the  contribution  of  ISO  9000  to  the  results  but  are  solely  a  “before  and  after” 
comparison. 

From  the  academic  perspective,  a  paper  written  by  Gavin  P.M.  Dick,  a  Senior 
Lecturer  at  Staffordshire  University  Business  School,  contends  that  a  proven  correlation 
exists  between  quality  and  business  perfonnance,  including  sales  growth,  financial 
returns  and  overall  company  performance.52  However,  in  Dick’s  opinion,  the  research 
evidence  does  not  support  a  link  between  improved  business  perfonnance  and  ISO  9000 
certification.  He  found  that  an  important  indicator  of  business  performance  gains 
achieved  by  certification  is  the  motivation  that  initially  led  to  the  pursuit  of  certification. 

“Companies  that  cited  customer  pressure  as  their  reason  for  pursuing  certification 
were  less  likely  to  report  improvements  than  those  which  gave  other  reasons  for  adopting 


5 1  BSI  case  studies  summary,  Richard  Lutman,  McLaughlin  Research  Corporation,  May  2004. 

52  Gavin  P.  M.  Dick,  P.  2. 
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QCert”53  (a  certified  quality  management  system).  This  was  echoed  in  the  Dallas 
Business  News  case  cited  previously  when  the  president  of  the  company  commented  on 
the  pursuit  of  ISO  certification  as  a  sales  and  marketing  tool:  “It’s  only  a  marketing  tool 
until  everyone  has  it...  for  companies  who  do  it  wrong  it’s  a  bunch  of  baloney.”54  Since 
at  least  half  of  the  companies  in  the  BSI  case  study  cited  some  form  of  customer  or 
market  pressure  as  a  motivating  factor,  “marketplace  coercion”  is  as  significant  a  factor 
as  the  desire  to  improve  for  many  companies  implementing  ISO  9000. 

Vanguard  Education,  a  United  Kingdom-based  company  that  markets  educational 
products  for  the  application  of  systems  thinking  to  improve  organizational  performance,55 
goes  a  step  further  with  their  opinion  of  ISO  9000  certification:  “registration  to  ISO  9000 
should  be  avoided.”5®  They  assert  that  the  use  of  ISO  9000  generates  a  mindset  of  rigid 
process  conformity  as  opposed  to  a  true  focus  on  product  quality,  process  improvement 
and  innovation.  However,  they  do  not  ignore  the  fact  that  ISO  9000  is  expanding  as  a 
requirement  to  do  business  in  many  sectors  and  offer  the  Vanguard  Standards  for 
“systems  thinking”  as  an  alternative  quality  system.  The  Vanguard  Standards  are  an 
extension  of  ISO  9000  and  were  developed  as  a  tool  for  learning  and  improvement,  as 
opposed  to  strict  conformity,  that  enables  achievement  of  ISO  9000  certification  with 
increased  emphasis  on  product  quality  and  customer  value. 

From  the  foregoing  discussion,  it  is  possible  to  conclude  that  ISO  9000  certifi¬ 
cation  can  provide  improved  business  performance  if  used  in  the  proper  perspective:  as  a 
management  tool  supported  by  the  appropriate  level  of  management  commitment  to 
improve  corporate  quality.  The  main  difference  between  ISO  9000  and  other  quality 
management  systems  is  the  fact  that  many  companies  and  government  agencies,  both 
domestic  and  international,  are  requiring  ISO  9000  certification  of  their  suppliers.  This 


53  ibid. 

54  Margaret  Allen,  “Polyguard  Bolsters  Quality,  Productivity  with  ISO  9000  System  Sysprocess,” 
Dallas  Business  Journal,  15  August  1997  [journal  on-line],  P.  4,  Available  at  http://www.biziournals.com/ 
4dlas/si^e^l997/Qt/ll/j^allfe:Lhlml.  Accessed  April  23,  2004. 

55  Vanguard  Educational  Website,  http://www.lean-service.com/sub2.asp.  Updated  10  November  2003. 

56  “The  Vanguard  Standards:  A  systems  thinkers  guide  to  interpretation  and  use  of  ISO  9000:2000,” 
P.2,  http://www.lean-service.com/3-l.asp.  Accessed  April  23,  2004. 
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trend  provides  an  advantage  for  using  ISO  9000  as  opposed  to  other  quality  systems  since 
it  serves  to  maximize  the  customer  base  of  an  organization. 

A  commonly  cited  disadvantage  of  using  ISO  9000  is  the  cost  and  time  require¬ 
ments  associated  with  defining,  documenting  and  maintaining  an  organization’s  quality 
policy,  procedures  and  work  instructions.57  The  commitment  of  overhead  resources  and 
allocation  of  management  time  necessary  to  adequately  define  the  processes  of  a  business 
unit  can  represent  a  significant  up-front  cost.  In  the  authors’  opinion,  the  expense  of  ISO 
9000  certification  results  in  a  largely  unquantifiable  retum-on-investment  for  organiza¬ 
tions  such  as  the  government  that  do  not  have  an  economically  quantifiable  business 
process. 

To  validate  this  assertion,  it  is  necessary  to  define  an  “economically  quantifiable 
business  process”  for  this  context.  Since  ISO  9000  originated  in  the  manufacturing 
sector,  we  will  begin  with  that  perspective.  For  a  manufacturer,  the  organization’s 
business  processes  generally  center  on  the  transformation  of  incoming  material  to  a 
refined  end  product  for  a  number  of  identical  end-items.  Each  step  of  an  effective 
process  will  be  definable  and  therefore  lends  itself  to  quantifiable  measurement.  Use  of  a 
variety  of  analysis  tools,  such  as  statistical  process  control,  can  provide  detailed  insight 
into  the  internal  operations  of  the  process  and,  in  so  doing,  facilitate  an  assessment  of  the 
efficiencies  of  process  components.  Assessment  of  process  components  is  then  used  to 
optimize  the  overall  efficiency  of  the  organization’s  end-to-end  process. 

For  an  organization  considering  ISO  9000  implementation,  the  availability  of 
process  analysis  data  can  significantly  contribute  to  the  initial  cost-benefit  analysis  to 
quantify  the  expense  of  certification  with  respect  to  the  benefits  to  be  gained. 
Estimations  in  overall  process  improvement  can  be  directly  translated  into  a  financial 
frame  of  reference,  providing  decision  makers  with  an  objective  measurement  of  the 


57  The  Business  Link  —  Business  Service  Center,  “What  is  ISO  9000?”  http://www.cbsc.org/ 
a  lb  erta  /tb  1 .  c  fm?  fn=i  so ,  Updated  11  March  2003,  Accessed  April  23,  2004. 
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returns  expected  from  the  investment  in  an  ISO  9000  quality  management  system.  Since 
most  corporate  analysis  tools  operate  within  the  financial  domain,  these  assessments 
work  well  within  that  context. 

However,  process  analysis  tools  depend  on  a  large  number  of  samples  to  establish 
statistical  significance  and  thereby  provide  a  reliable  measure  of  effectiveness  for 
changes  to  organizational  processes.  For  most  government  activities,  this  metric  is  not 
valid  since  governments  do  not  normally  engage  in  manufacturing-like  enterprises;  those 
activities  are  contracted  to  private  industry.  An  exception  to  this  observation  can  be 
found  in  operations  such  as  military  depots,  where  items  are  refurbished  using  a  repetitive 
process  that  lends  itself  to  process  analysis.  However,  the  current  trend  is  to  outsource 
this  type  of  work  to  industry  wherever  it  is  economically  advantageous. 

Most  government  activities  operate  in  a  more  ambiguous  environment  where 
organizational  inputs  and  outputs  are  not  often  directly  traceable  through  defined  and 
consistent  processes.  Even  those  activities  that  have  a  defined  process  may  have  a  high 
degree  of  variability  in  each  sample.  For  example,  the  processing  of  income  tax  returns 
by  the  Internal  Revenue  Service  (IRS)  has  a  defined  and  structured  process.  However, 
because  of  the  complexity  and  dynamic  nature  of  tax  laws,  it  would  be  difficult  to  derive 
a  statistically  valid  assessment  of  the  efficiencies  of  that  process.  The  same  argument 
could  be  made  for  the  processing  of  Social  Security  claims  or  veterans’  benefits,  since 
each  example  operates  in  a  complex  statutory  and  policy  environment  serving  individual 
needs. 

Similarly,  the  processes  in  a  U.S.  government  military  research  and  development 
organization  have  a  high  degree  of  variability  resulting  from  the  inherently  unpredictable 
nature  of  research  and  development  and  dynamic  requirements  of  the  current  military 
threat  environment.  For  example,  attempting  to  apply  statistical  process  measurement 
tools  to  the  development  of  a  piece  of  technology,  such  as  a  digital  signal  processor,  for 
different  applications  would  have  little  value  because  of  the  variability  of  the  require¬ 
ments  and  operating  environment  for  each  application.  Statistical  tools  could  be  applied 
to  the  production  of  multiple  units  of  a  single  design  with  a  high  degree  of  effectiveness, 

but  this  function  would  normally  be  contracted  to  industry. 
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The  foregoing  analysis  draws  to  a  fundamental  difference  between  public  and 
private  sector  enterprises:  the  repeatability,  and  therefore  profitability,  of  the  organiza¬ 
tion’s  processes  as  a  function  of  the  business  base.  For  instance,  the  RSDBU  is  retained 
as  a  government  enterprise  because  the  business  base  is  highly  variable  and  unpre¬ 
dictable,  thereby  making  it  unprofitable  for  industry  to  maintain  a  robust  base  of  available 
providers.  The  variability  of  the  business  base  leads  to  organizational  processes  that 
cannot  be  reliably  statistically  quantified,  which  serves  as  a  disincentive  for  industry  to 
engage  in  this  function.  Thus,  the  majority  of  government  activity  in  general,  and 
military  research  and  development  specifically,  falls  into  a  business  area  that  is  not 
economically  quantifiable. 

Hence,  the  impact  of  a  documented  quality  management  system  on  the  organiza¬ 
tional  output  of  a  government  enterprise  will  be  difficult  to  reliably  predict  prior  to 
making  the  investment  due  to  the  limitations  of  statistical  process  measurement  for 
government  functions.  Accurately  measuring  the  impact  of  a  quality  system  on  a 
government  organization’s  performance  metrics  can  be  equally  difficult  once  it  has  been 
established  for  this  same  reason.  Gross  measures  may  be  derived  based  on  “before  and 
after”  comparisons,  but  attributing  detailed  numerical  quantities  such  as  a  reduction  in 
operational  costs  or  a  decrease  in  cycle  time  to  the  predicted  performance  of  a  quality 
system  would  have  a  highly  subjective  interpretation. 

Corporate  and  government  cost-benefit  methodologies  are  generally  geared  to  the 
realm  of  capital  purchase  decisions  based  on  assumptions  of  increased  productivity 
and/or  reduced  costs  associated  with  an  investment.  Analysis  methodologies,  including 
net  present  value,  internal  rate  of  return,  payback  period,  least  cost  and  similar 
approaches,58  rely  on  numerical  quantifications  of  expected  outcomes  and  can  provide 
significant  value  in  framing  a  decision.  However,  these  approaches  should  be  used  with 
extreme  caution  when  evaluating  investment  in  a  quality  management  system  because  of 
the  prevalence  of  the  human  element  in  realizing  gains  from  implementation.  Estimating 


58  Naval  Postgraduate  School  of  Business  and  Public  Policy,  Course  material  from  MN3155,  Financial 
Management  for  Acquisition  Managers,  Winter  Qrtr  2000,  P.  20-30 
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the  gains  to  be  achieved  before  implementing  the  system  is  at  best  a  “what  if’  scenario 
process  of  correlating  potential  gains  to  a  payback  period  for  process  documentation  and 
certification  costs. 

Based  on  the  assumption  that  most  cost-benefit  analyses  for  ISO  9000  imple¬ 
mentation  for  a  government  organization  will  be  of  limited  utility,  how  does  ISO  9000 
offer  an  advantage?  Perhaps  the  best  answer  to  this  question  is  found  in  the  fact  that  an 
organization  that  goes  through  an  ISO  9000  process  documentation  initiative  is  forced  to 
examine  all  of  the  internal  business  process  steps  required  to  transform  organizational 
inputs  to  organizational  outputs.  An  effective  management  team  will  use  the  opportunity 
to  examine  organizational  process  in  search  of  muda,  or  waste,  using  the  same 
philosophy  as  Japanese  lean  producers59  that  seek  to  eliminate  anything  that  does  not 
directly  add  value  to  the  end  product. 

ISO  9000  registration  efforts  may  be  used  as  an  opportunity  to  evaluate  and 
optimize  processes  as  opposed  to  simply  documenting  the  existing  ways  of  doing 
business.  Those  parts  of  organizational  processes  that  do  not  add  value  to  the  final  output 
should  be  eliminated;  to  do  otherwise  is  simply  the  act  of  documenting  a  suboptimal 
process.  While  this  forced  introspection  could  be  accomplished  without  the  use  of  ISO 
9000,  the  standard  provides  a  useful  framework  to  ensure  that  a  comprehensive 
examination  is  conducted. 

The  use  of  ISO  9000  also  suffers  from  the  lack  of  a  centralized  registration 
recording  structure.  According  to  the  disclaimer  on  the  ISO  organization  website:  “there 
is  no  ‘official  central  database’  of  ISO  9001  or  ISO  14001  certificates  and  ISO  does  not 
hold  lists  of  names  or  any  information  about  certified  organizations.”60  The  structure  of 
the  ISO  certification  process  is  such  that  independent  registrars  are  the  only  bodies  that 
issue  certificates.  Therefore,  statistics  on  certifications  can  be  difficult  to  track,  since 


59  Nick  Rich,  “An  Executive  Guide  to  Lean  Thinking,”  Deloitte  &  Touche  Senior  Executive  Series  of 
Briefing  Papers  [Deloitte  &  Touche  Senior  Research  Fellow  Lean  Organisation  Research  Centre,  Cardiff 
Business  School  ],  P.  5-6,  http://www.deloitte.com/dtt/cda/doc/content/lean  briefing!  1  )(T).pdf.  Modified 
6  June  2002, ,  Accessed  April  23,  2004. 

60  International  Organization  for  Standardization,  “Statistics  on  certifications  worldwide,”  1st  para., 
http://www.iso.ch/iso/en/xsite/contact/01enquiryservice/01  llS09000/certification/stats.html,  Modified 
8  March  2004,  Accessed  April  23,  2004. 
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registrars  appear  to  have  different  styles  when  it  comes  to  providing  certification  infor¬ 
mation.  This  is  exemplified  by  the  apparently  differing  applications  of  the  Standard 
Industrial  Classification  (SIC)  codes  applied  to  ISO  certified  organizations. 

The  SIC  system  of  the  U.S.  Department  of  Labor  Occupational  Safety  and  Health 
Administration61  serves  as  the  basis  for  categorizing  the  type  of  business  in  which  an 
organization  primarily  participates.  There  is  a  separate  British  SIC  system  used  by  many 
U.K.  registrars  that  does  not  directly  correlate  to  the  U.S.  system.  Also,  no  categorization 
of  ISO  registrants  exists  on  the  basis  of  being  a  public  versus  a  government  organization, 
although  there  is  a  section  of  the  SIC  codes  devoted  to  “Public  Administration”  where 
one  could  reasonably  expect  to  find  government  organizations. 

An  analysis  of  the  10  ISO  9001  registered  U.S.  Navy  organizations,  as  recorded 
in  the  Quality  Digest  magazine  ISO  9000  database,62  reveals  that  they  are  classified 
across  numerous  SIC  codes,  including  subcategories  of  manufacturing,  services,  trans¬ 
portation,  electric,  gas  and  sanitary  services  and  even  wholesale/retail  trades.  While 
some  of  this  may  be  due  to  differing  registrar  approaches,  the  disclaimer  on  the  database 
also  notes:  “[w]e  have  found  that  industry  classification  codes  are  often  input 
incorrectly.”63  Perhaps  this  is  an  area  where  the  ISO  registrars  and  standards  develop¬ 
ment  bodies  could  apply  some  of  their  own  wisdom.  The  development  of  standardized 
processes  for  recording  certification  data  in  a  common  repository  using  consistent 
reference  styles  could  significantly  improve  conformity.  The  composition  of  the  Quality 
Digest  ISO  database  appears  to  have  a  notable  lack  of  conformity  in  terms  of  level  of 
detail  as  well  as  classification  of  entries. 

In  summary,  ISO  9000  offers  a  defined  methodology  of  process  documentation 
that  may  be  used  for  organizational  performance  improvement  if  applied  correctly. 
However,  ISO  9000  can  initially  be  an  expensive  proposition  because  of  the  scope  of  the 
effort  involved  in  process  documentation,  and  may  be  difficult  to  financially  justify  from 


61  U.S.  Department  of  Labor,  Standard  Industrial  Classification  (SIC)  System  Search, 
http://www.osha.gov/pls/imis/sicsearch.html.  Accessed  April  23,  2004. 

62  Quality  Digest,  ISO  9000  Database,  http://www.qualitvdigest.com/iso9000database.shtml.  Accessed 
May  22,  2004. 

63  ibid. 
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a  return-on-investment  perspective  unless  a  rigorous  cost  analysis  system  is  already  in 
place  and  functioning  efficiently.  If  a  cost-benefit  justification  is  necessary,  the  business 
of  the  organization  should  also  be  such  that  “before  and  after”  comparisons  are  realistic 
and  based  on  consistently  measurable  quantities.  Cost  benefit  analyses  should  also 
reflect  that  once  the  initial  certification  is  achieved,  resources  must  be  devoted  to  main¬ 
taining  and  improving  the  quality  system. 

Therefore,  ISO  9000  may  not  be  a  significantly  better  management  tool  than 
many  of  the  other  quality  management  schemas  that  are  widely  available,  but  ISO  9000 
comes  with  the  added  benefit  of  international  recognition  that  allows  an  organization  to 
maximize  its  customer  base.  If  a  management  team  is  truly  motivated  to  improve  organ¬ 
izational  performance,  ISO  9000  can  be  a  valuable  tool  to  assess  the  complete  process. 
However,  the  final  decision  on  a  quality  management  system  should  be  based  on  the 
compatibility  of  the  tool  to  the  organization’s  goals  and  operating  methodologies, 
combined  with  the  demands  of  the  market  in  which  the  organization  operates.  If  “market 
coercion”  is  a  factor,  that  should  be  understood,  but  ISO  9000  will  provide  the  most 
benefit  if  it  is  used  as  a  tool  to  improve  organizational  performance,  as  opposed  to  a 
minimalist  approach  intended  primarily  to  mollify  customers. 

C.  SIMILARITIES  AND  DIFFERENCES  BETWEEN  ISO  APPLICATIONS 

ISO  9000  provides  a  standardized  approach  for  the  establishment  of  a  quality 
management  system  that  can  be  used  across  virtually  any  organization.  However,  as 
previously  discussed,  cost-benefit  analyses  to  justify  the  initial  investment  may  not  be 
reliable;  an  organization  may,  nevertheless,  proceed  with  ISO  certification  efforts,  based 
on  a  rational  desire  to  improve  perfonnance,  using  ISO  9000  as  an  organizational 
analysis  tool. 

From  the  RSDBU  perspective,  it  is  impossible  to  definitively  quantify  the  value 
of  the  ISO  9001:2000  Quality  Management  System  (QMS)  implemented  based  on 
objectively  measurable  criteria  such  as  financial  performance  or  product  quality 
measurements.  Due  to  the  wide  variance  amongst  RSDBU  programs  conducted  both 
historically  and  at  any  given  time,  an  objective  “before  and  after”  analysis  is  not  realistic. 
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Returning  to  the  primary  motivations  for  RSDBU  quality  management  certification 
demonstrates  that  these  goals  are  difficult  to  measure  objectively: 

•  Instill  discipline  in  planning  and  cost  estimates 

•  Better  understanding  of  technical  work  being  performed  across  the 
organization 

•  Ability  to  bid  on  FMS  work  requiring  ISO  9000  certification 

•  Documentation  of  lessons  learned 

Even  for  those  items  for  which  an  objective  measure  can  be  devised,  the  assign¬ 
ment  of  the  proportion  of  the  measured  result  that  is  attributable  to  ISO  9000  certification 
will  always  be  debatable  in  any  analysis.  To  expand  on  this  point,  a  cursory  review  of 
possible  analysis  methods  to  prove  that  the  Code  70  objectives  for  the  RSDBU  have  been 
met  will  illustrate  the  difficulty  of  deriving  a  quantifiable  return  on  investment: 

Instill  discipline  in  planning  and  cost  estimates:  This  aspect  could  be  measured 
by  the  number  of  programs  that  achieve  cost,  performance,  and  schedule  benchmarks 
versus  those  that  do  not  or  through  trend  analysis  of  RSDBU  program  execution 
performance.  However,  as  illustrated  in  the  AMSWR  and  ADAS  examples  of  Chapter  2, 
ISO  9000  has  varying  degrees  of  effectiveness  on  program  performance  issues  that 
cannot  be  specifically  attributed  to  the  influence  of  a  quality  management  system. 

Better  understanding  of  technical  work  being  performed  across  the 
organization:  “understanding”  is  a  nebulous  tenn  to  measure  without  applying  some 
form  of  academic  rigor,  which  in  this  case  would  be  complex  and  of  limited  utility. 
Subjective  and  primarily  anecdotal  estimation  would  be  the  primary  measurement 
method. 

Ability  to  bid  on  FMS  work  requiring  ISO  9000  certification:  With  the 
achievement  of  ISO  9000  certification,  the  RSDBU  attained  the  ability  to  bid  on  FMS 
work  requiring  ISO  9000  certified  offerors,  so  in  that  context  the  RSDBU  quality  man¬ 
agement  system  could  be  considered  to  have  met  the  desired  objective.  However,  since 
certification,  the  RSDBU  has  not  actually  gained  any  FMS  work,  which  could  be  inter- 


57 


preted  as  a  failure  of  the  system  implemented,  but  is  more  likely  due  to  the  vagaries  of 
FMS  work  volume.  However,  due  to  the  inherent  complexities  of  the  FMS  market,  an 
attempt  to  allocate  success  or  failure  to  either  assertion  would  be  of  no  value. 

Documentation  of  lessons  learned:  The  RSDBU  quality  management  system 
does  not  provide  a  defined  methodology  for  documenting  lessons  learned  other  than 
mandating  consistent  program  documentation  and  review  requirements  and  continuous 
improvement  initiatives.  Eventually  it  may  be  possible  to  derive  an  estimate  of  cost 

avoidance  if  documentation  of  a  previous  program  proves  to  be  useful  in  the  conduct  of 
future  business,  but  the  actual  dollar  value  would  ultimately  be  a  highly  subjective 
estimate. 

These  examples  illustrate  what,  in  the  authors’  opinion,  is  the  biggest  disad¬ 
vantage  of  using  ISO  9000  for  an  organization  such  as  the  RSDBU:  the  inability  to 
objectively  measure  and  quantify  the  benefits  of  a  documented  quality  management 
system.  In  a  commercial  manufacturing  setting,  statistical  production  metrics  can  be 
compared  in  a  ‘before  and  after’  context  to  determine  if  defect  rates,  station  cycle  times 
or  other  easily  quantifiable  measurements  have  changed.  Those  reductions  can  be 
translated  into  approximate  dollar  values,  and,  therefore,  a  quantified  change  in  per¬ 
formance  can  be  described.  For  the  RSDBU,  this  is  not  possible  because  the  broad 
variation  and  low  quantity  of  output  products  invalidates  statistical  significance. 

This  aspect  makes  it  difficult  to  justify  the  initial  expense  of  process  documenta¬ 
tion  and  equally  as  hard  to  justify  the  continued  expense  of  process  maintenance  for 
organizations  such  as  the  RSDBU.  Whether  the  product  is  program  management,  system 
engineering  services  or  actual  hardware  and  software  deliverables,  the  fact  that  each  task 
differs  significantly  from  either  previous  or  ongoing  tasks  effectively  eliminates  an 
objectively  analytical  basis  for  comparison.  It  is  counterproductive  to  attempt  to  quanti¬ 
tatively  assess  the  benefits  of  certification  from  a  cost  analysis  perspective  by  comparing 
a  “pre-ISO”  and  “post-ISO”  RSDBU  program.  This  forces  an  organization  whose 
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members  are  almost  exclusively  engineers  and  scientists  to  accept  a  subjective  assess¬ 
ment  of  certification  benefits,  which  tends  to  be  inconsistent  with  the  organizational 
culture. 

Other  government  organizations  have  applied  the  ISO  9001  in  manners  both 
similar  to  and  different  from  the  RSDBU.  The  10  ISO  9001  registered  U.S.  Navy 
organizations  as  recorded  in  the  Quality  Digest  magazine  ISO  9000  database  comprise 
both  logistics-focused  organizations  as  well  as  RDT&E  activities.  Navy  ISO  9001 
registrations  (including  both  1994  and  2000  versions)  include  naval  aviation  depots,  the 

Naval  Medical  Education  and  Training  Command  and  components  of  the  Naval  Surface 

Warfare  Center  and  Naval  Undersea  Warfare  Center.64  The  scope  of  their  certifications 
includes: 

•  Aircraft  maintenance,  repair,  overhaul,  and  modification  services 

•  Technical  library  services 

•  Healthcare  acquisition  and  logistics  support  to  Navy  medical  and  dental 
units 

•  Design  and  development  of  training  films,  graphics,  and  curriculum  for 
the  Naval  Health  System  worldwide 

•  Aircraft  support  services  (paint  &  flight  test) 

•  Engineering,  logistics,  maintenance,  and  program  management  for 
electronics,  electronic  warfare,  and  ordnance 

•  Hardware  and  software  design,  engineering,  parts  procurement, 
production,  integration,  check  out,  test,  of  the  AN/UPX-36  (V)6  IFF 
System 

•  Acquisition  program  support 

•  Engineering  of  polymers,  composites,  metals  and  ceramics 


64  Ibid.,  16  May  2004. 
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Nondestructive  evaluation 


• 

•  Corrosion  and  materials  protection  technologies  and  materials  engineering 

•  Laboratory  testing  services  for  calibration  and  verification 

•  Research  and  development,  test  and  evaluation,  fleet  support,  test 
facilities,  and  in-service  engineering  for  surface  and  undersea  vehicle  hull, 
mechanical,  and  electrical  systems  and  propulsions,  and  logistics  research 
and  development 

•  Design  services  for  and  provision  of  maintenance  and  repair  services 

•  Design,  development,  integration,  testing  and  installation  of  undersea 
warfare  training,  test  and  evaluation  systems65 

This  list  demonstrates  that  the  Navy  business  units  that  have  become  certified 
differ  in  focus,  from  narrowly  defined  areas  such  as  aircraft  logistics  support,  to  more 
broadly  scoped  areas  of  research  and  development  for  a  class  of  systems.  In  these 
instances,  ISO  9001  was  applied  at  an  activity  or  organizational  level  with  a  reasonably 
well-defined  focus  that  allows  a  consistency  of  processes  that  facilitates  the  achievement 
of  the  organization’s  mission.  The  flexibility  of  the  ISO  9001:2000  standard  is  the 
primary  characteristic  that  enables  organizations  operating  within  this  spectrum  of 
specialization  to  be  certified  under  a  common  system. 

As  shown  above,  U.S.  Navy  organizations  implemented  ISO  9001  in  order  to 
achieve  better  cost  efficiency  and  quality  perfonnance  within  a  fairly  narrowly  defined 
scope  of  endeavor.  For  example,  naval  aviation  depots  perform  a  defined  and  consistent 
scope  of  tasks  in  the  process  of  refurbishing  aircraft.  ISO  9001  offers  a  method  of 
standardized  process  documentation  that  facilitates  maximizing  the  efficiency  of  these 
organizations  as  well  as  providing  a  “brand  name”  quality  system  that  may  also  provide 
advantages  during  the  conduct  of  outsourcing  (A-76)  studies  to  fulfill  quality  assurance 
requirements  as  well  as  BRAC  (Base  Realignment  and  Closure)  analyses. 


65  ibid. 
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As  discussed  in  earlier  chapters,  NUWC  Division  Newport,  as  a  Navy  Working 
Capital  Fund  activity,  has  a  different  incentive  base  than  mission-funded  Navy  ISO  9000 
certified  organizations:  to  remain  viable  based  on  continued  customer  funding.  ISO 
9001:2000  certification  for  the  RSDBU  was  perceived  as  a  tool  that  would  enable  the 
RSDBU  to  better  support  a  business  base  that  retains  the  core  equity  of  undersea  range 
development  while  simultaneously  contributing  to  the  attainment  of  NWCF  financial 
goals.  Process  standardization  can  contribute  to  meeting  these  objectives  by  improving 
internal  processes  to  maximize  value  to  the  customer  and  thereby  increase  the  likelihood 
of  maintaining  or  improving  the  business  base.  However,  because  process  standardi¬ 
zation  was  not  a  broadly-based  organizational  imperative,  the  approach  used  was 
selective  certification  of  an  individual  business  unit. 

In  contrast,  an  organization  such  as  the  National  Aeronautics  and  Space 
Administration  (NASA)  chose  to  approach  ISO  9000  certification  for  the  entire 
enterprise,  including  its  headquarters  and  1 1  research  and  development  centers.  In  a 
1996  letter  to  Officials-in-Charge  of  Headquarters  Offices,  Directors  of  NASA  Field 
Installations,  and  the  Director  of  the  Jet  Propulsion  Laboratory  (JPL),  NASA 
Administrator  Daniel  S.  Goldin  wrote,  “We  are  leaders  in  the  world  of  science  and 
technology.  We  must  also  be  leaders  in  the  world  of  quality.  To  this  end,  I  am  requiring 
that  the  Agency  be  third-party  certified  in  our  key  processes,  by  an  internationally 
recognized  registrar,  to  ISO  9001.  This  commitment  applies  to  all  Centers  and 

Headquarters. ”66 

NASA  operates  under  a  different  set  of  incentives  than  a  Defense  Working 
Capital  Fund  activity  and,  consequently,  had  differing  motivations  from  the  RSDBU  and 
thereby  chose  a  different  path  to  certification.  The  level  of  public  and  congressional 
visibility  received  by  NASA  quality  issues  was  likely  the  greatest  single  motivating 
factor  in  the  institution  of  an  enterprise-wide  quality  management  system.  The 


66  Frederick  D.  Gregory,  Associate  Administrator  for  Safety  and  Mission  Assurance  [signatory],  NASA 
Headquarters,  “Program  Commitment  Agreement,  NASA  Registration  to  the  United  States  Version  of  the 
International  Organization  for  Standardization’s  9000  Series  Standard,”  9  July  1999, 
http://www.hq.nasa.gov/office/codeq/doctree/isopca.pdf.  Accessed  May  22,  2004. 
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enterprise-wide  approach  also  serves  to  deflect  any  public  or  political  scrutiny  that  any 
part  of  NASA  would  receive  had  it  been  exempted  from  the  certification  effort. 

However,  the  NASA  ISO  9001  certification  effort  apparently  has  not  been  com¬ 
pletely  successful  in  fulfilling  organizational  quality  objectives  since  the  completion  of 
certification  of  all  sites  on  September  24,  1999. 67  Fredrick  Gregory,  NASA  Deputy 
Administrator,  wrote  in  a  December  12,  2002  letter:  “[rjecent  experience,  as  expressed  in 
Freedom  to  Manage  (F2M)  inputs  from  several  Centers  and  Headquarters  Offices, 
indicates  that  ISO  9001  may  not  be  the  only  or  the  best  tool  for  every  situation.”^  On 
April  17,  2003,  NASA  changed  their  Management  System  Policy69  to  remove  the  ISO 
9001  certification  as  a  mandate,  substituting  their  own  internally  developed  set  of 
standards.  NASA  Policy  Directive  (NPD)  1280.1  is  intended  to: 

. .  .provide  Center  Directors  with  both  the  responsibility  and  flexibility  to 
implement  management  systems  commensurate  with  the  risks  associated 
with  the  activities  under  their  care.  We  anticipate  that  most  Centers  will 
retain  ISO  9001  as  their  means  of  complying  with  this  new  NPD. 
However,  Centers  will  have  freedom  to  experiment  with  or  move  to  other 
systems,  as  long  as  the  same  minimum  standards  are  met,  such  as  third 
party  validation.-^ 

This  change  in  policy  could  lead  to  the  conclusion  that  ISO  9001  may  not  be  a 
single  point  solution  for  all  organizations,  particularly  large  public  enterprises  with 
complex  organizational  missions.  Even  with  the  inherent  flexibility  of  the  ISO  9001 
standard,  its  implementation  must  be  consistent  with  the  requirements  of  the  organization 
to  ensure  it  functions  as  an  effective  management  tool. 

The  foregoing  comparison  illustrates  the  opposite  extreme  of  ISO  9001  applica¬ 
tion  from  the  RSDBU  where  ISO  9001  was  applied  at  the  enterprise  level.  The 


67  Sonja  Alexander,  “NASA  Achieves  ISO  9001  Registration  at  All  Sites,”  24  September  1999, 
http://articles.fmdarticles.eom/p/artieles/mi  pasa/is  /ai  366684083,  Accessed  May  22,  2004. 

68  AD/Deputy  Administrator,  “ISO  9001  and  Beyond”  [letter],  To  Offtcials-in-Charge  of  Headquarters 
Offices  Directors,  NASA  Centers,  12  December  2002,  P.  1,  http://www.hq.nasa.gov/iso/ 

12  02  gregory.doc.  Accessed  May  22,  2004. 

69  NASA  Quality  Management  Systems,  “Welcome  to  the  Agency  Quality  Management  System 
Page!,”  para.  1,  http://www.hq.nasa.gov/iso/.  Accessed  May  22,  2004. 

7 6  AD/Deputy  Administrator,  “ISO  9001  and  Beyond”  [letter],  12  December  2002,  P.  1, 
http://www.hq.nasa.gov/iso/12  02  gregory.doc.  Accessed  May  22,  2004. 
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previously  discussed  Navy  implementations  represent  a  middle  ground  where  ISO  9001 
was  applied  at  the  activity  level  for  organizations,  such  as  naval  aviation  depots,  that  have 
more  specifically  defined  organizational  functions.  The  NASA  retreat  from  an  ISO  9001 
certification  approach  for  the  total  organization  reveals  that  ISO  can  become  unwieldy 
and  less  effective  if  the  scope  of  the  certification  is  expanded  across  a  very  broad  field  of 
endeavor,  such  as  space  flight.  While  the  RSDBU  approach  may  be  considered  a  “micro 
application”  of  ISO  9001  in  comparison,  it  demonstrates  that  this  approach  is  likely  the 
best  for  organizations  that  do  not  have  clear  cultural  or  environmental  drivers  that  justify 
the  time  and  expense  associated  with  a  large-scale  certification  effort. 

D.  EFFECTS  ON  COMPETITIVE  ADVANTAGE  FOR  THE  RSDBU 

To  analyze  this  question,  it  is  necessary  to  first  define  “competitive  advantage” 
within  the  context  of  the  RSDBU  and  its  parent  organization,  the  Naval  Undersea 
Warfare  Center.  The  ongoing  realignment  of  the  NAVSEA  warfare  centers  that  includes 
the  six  divisions  of  the  Naval  Surface  Warfare  Center  and  the  two  NUWC  divisions  is 
intended  to  create  an  environment  which  is  non-competitive.  The  NSWC  and  NUWC 
organizations  providing  the  core  equities  necessary  to  support  the  NAVSEA  mission  are 
defined  as  unique  national  assets  that  are  not  duplicated  in  industry  and  are  required  to 
preserve  national  military  capabilities.  The  NAVSEA  realignment  is  intended  to  provide 
a  structure  that  identifies  and  adjusts  the  workforce  and  associated  physical  infrastructure 
necessary  to  efficiently  support  these  core  equities  within  the  context  of  Navy  strategic 
planning,  which  is  currently  embodied  in  the  CNO’s  Sea  Power  21  vision.71 

However,  at  the  activity  level,  the  primary  measure  of  effectiveness  remains  Net 
Operating  Result  (NOR),  based  on  the  annual  financial  performance  of  the  organization. 
NOR  is  the  public  organization  equivalent  of  the  profit/loss  metric  used  in  commercial 
industry;  the  main  difference  being  that  a  government  organization  does  not  seek  to  make 
a  profit,  but  balance  income  with  expenses  thereby  ideally  achieving  a  NOR  of  zero.  The 
NOR  metric  is  driven  by  the  fact  that  the  NAVSEA  field  activities  operate  in  the  NWCF 
construct  that  is  intended  to  identify  the  relative  efficiencies  of  organizations.  Activities 


71  NUWCDIVNPT,  NUWC  Concept  of  Operations  (CONOPS).  25  January  2004,  P.  2. 
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generating  a  negative  NOR,  which  is  effectively  a  “loss,”  can  be  classified  as  either 
inefficient  or  having  overcapacity  for  the  current  needs  of  the  NAVSEA  mission. 

Within  this  external  operating  environment,  the  NAVSEA  realignment  process, 
combined  with  the  NWCF  constraints,  incentivizes  organizations  to  demonstrate  their 
value-added  through  both  technical  proficiency  and  business  efficiencies.  This  is 
reflected  by  one  of  the  many  duties  of  the  NUWC  Technical  Director  (TD),  who  has  a 
responsibility  to  “ensure  that  a  robust  technical  capability  is  maintained  within  the  Navy 
without  duplication  and/or  competing  capabilities.”72  An  additional  duty  of  the  TD  is  “to 
ensure  that  NUWC’s  workforce  is  balanced  with  the  Navy’s  workload,”73  which  implies 
both  expanding  the  workforce  where  additional  capacity  is  required,  or  reducing  or 
realigning  the  existing  workforce  where  overcapacity  is  identified. 

Nonetheless,  a  “competitive  advantage”  within  this  context  represents  an  organi¬ 
zation’s  ability  to  demonstrate  that  it  is  technically  proficient  and  economically  efficient 
in  the  performance  of  its  mission  in  support  of  NAVSEA  objectives,  whether  duplication 
exists  or  not.  If  there  is  duplication  in  the  fonn  of  internal  overcapacity  at  the  warfare 
center  level  or  external  overcapacity  at  the  NAVSEA  corporate  level,  senior  leadership 
will  detennine  how  each  issue  is  resolved  using  a  defined  methodology  for  work 
assignment.  Since  this  methodology  is  structured  to  align  the  workforce  along  roles  and 
missions  of  each  organization,  an  ISO  9001  certified  quality  system  will  not  be  an 
attribute  evaluated  in  the  decision  process. 

However,  it  is  incumbent  on  each  organization  to  optimize  their  operations  to 
maximize  quality  and  efficiency  as  a  mandate  of  public  service.  In  the  case  of  the 
RSDBU,  having  an  ISO  9001:2000  certified  quality  system  is  one  method  to  demonstrate 
a  commitment  to  both  quality  and  efficiency.  The  quality  system  serves  as  both  an 
organizational  credential  and  a  methodology  to  provide  greater  transparency  to  RSDBU 
customers  into  how  their  projects  will  be  executed. 


72  Ibid.,  P.  30. 

73  Ibid.,  P.  1. 
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The  preceding  analysis  demonstrates  that  an  ISO  9001:2000  certified  quality 
management  system  does  not  serve  to  provide  a  competitive  advantage  to  the  RSDBU 
within  the  context  of  their  operating  environment,  since  it  is  a  broader  organizational 
objective  to  have  non-duplicative  and  non-competitive  capabilities.  With  respect  to 
actual  RSDBU  financial  perfonnance,  it  cannot  be  shown  that  ISO  9001:2000  certifica¬ 
tion  has  had  a  definitive  effect  on  increasing  funding  or  preventing  the  loss  of  funding. 

As  previously  discussed,  the  primary  financial  incentive  for  achieving  ISO  9001 
certification  for  the  RSDBU  was  to  enable  pursuit  of  FMS  work.  Since  the  RSDBU  is 
not  currently  engaging  in  any  FMS  work,  it  cannot  be  demonstrated  that  funding  has 
increased  through  this  avenue.  Similarly,  none  of  the  current  RSDBU  customers  have 
formally  made  ISO  9000  certification  a  precondition  for  providing  new  tasking  or 
preserving  existing  tasking,  so  it  also  cannot  be  shown  that  ISO  9000  has  directly  con¬ 
tributed  to  the  expansion  or  preservation  of  the  RSDBU  business  base.  Therefore,  the 
merits  of  the  RSDBU  quality  management  system  must  be  determined  as  a  subjective 
assessment  as  to  whether  ISO  9001:2000  provides  benefit  through  improvement  of 
internal  processes  that  maximize  value  to  the  customer. 
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IV.  WOULD  ALTERNATIVE  SYSTEMS  WORK  BETTER? 


A.  INTRODUCTION 

This  chapter  discusses  the  advantages  and  disadvantages  of  alternate  quality  man¬ 
agement  systems  and  processes,  and  whether  any  of  these  alternate  systems  might  be 
more  appropriate  for  DoD  research  and  development  organizations. 

B.  ALTERNATE  STANDARDS  FOR  R&D  ORGANIZATIONS 

Research  and  development,  by  its  nature,  is  difficult  to  segment  into  consistent 
processes  that  can  be  measured  and  controlled.  The  processes  involved  with  R&D  are 
highly-coupled  with  the  interim  results  (i.e.,  the  ‘product’)  of  the  R&D  itself.  The 
creation  of  new  applications,  technologies  and  ideas  involves  a  significantly  greater  level 
of  risk  from  a  financial  standpoint,  since  it  is  challenging  to  accurately  predict  the 
resources  that  will  be  required  to  achieve  the  next  breakthrough,  or  even  incremental 
improvement.  Therefore,  the  selection  of  a  quality  management  system  for  a  research 
and  development  environment  must  reflect  these  realities  and  be  suitable  for  wide 
variations  in  process  outputs. 

The  disposition  of  the  people  that  are  generally  employed  in  research  and  devel¬ 
opment  environments  is  also  often  not  conducive  to  the  implementation  or  mandate  of 
process  confonnance.  Scientists  and  engineers  who  prefer  the  structure  of  scientific  and 
engineering  disciplines  tend  to  be  less  than  fond  of  processes  outside  those  disciplines 
that  interfere  with  the  creative  process  by  implementing  boundaries  of  schedule  or 
resources.  It  is  questionable  that  Dr.  J.  Robert  Oppenheimer  would  have  been  as 
effective,  if  the  Manhattan  Project  had  been  ISO  9000  certified. 

However,  today’s  DoD  research  and  development  organizations  do  not  operate 
under  the  same  external  imperatives  as  the  Manhattan  Project,  which  emphasized 
delivery  speed  above  all  else  during  wartime.  The  number  of  people  involved  in  the  DoD 
research  and  development  process  is  also  significantly  greater,  thereby,  incurring  a 
substantial  financial  resource  requirement  and  commensurate  need  to  operate  efficiently 
and  in  a  manner  that  maximizes  value  to  the  taxpayer.  The  current  DoD  environment  can 
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be  characterized  as  having  efficiency  in  cost  and  schedule  as  the  driving  external 
imperatives,  based  on  the  ongoing  economic  and  political  climate  of  constrained  budgets 
and  increased  military  operations  associated  with  the  War  on  Terror. 

Hence,  the  use  of  a  quality  management  system  within  DoD  research  and  devel¬ 
opment  organizations  serves  to  demonstrate  a  commitment  by  organizational  leadership 
to  achieve  the  current  external  imperative  of  efficiencies.  The  actual  efficiencies 
achieved  will  be  determined  by  the  relevance  of  the  implementation  to  organizational 
work  processes  and  continuing  senior  management  commitment  to  using,  maintaining 
and  improving  the  system  implemented.  The  use  of  a  commercially-developed  standard 
further  serves  to  support  the  objectives  of  acquisition  reform  and  conversion  to  com¬ 
mercial  standards  begun  by  Secretary  of  Defense  Perry  in  1994.  As  previously  stated,  the 
selection  of  that  standard  should  be  based  on  the  goals  and  operating  environment  of  the 
organization. 

As  can  be  seen  by  the  NASA  example  discussed  in  Chapter  3,  ISO  9001  was 
initially  selected  as  the  best  fit  for  that  organization,  but  management  policy  was  later 
modified  to  allow  for  greater  flexibility  at  the  discretion  of  organizational  component 
management.  This  brings  out  another  point:  that  the  processes  of  an  organization  should 
be  periodically  revisited  to  ensure  that  they  are  in  consonance  with  the  current  and 
projected  operational  environment.  The  continuance  of  a  program  simply  because  it 
exists  has  proven  to  be  an  unsound  course  of  action  in  many  past  cases. 

Based  on  the  examples  of  quality  management  systems  discussed  previously, 
there  is  no  specific  system  that  is  definitively  more  appropriate  for  a  DoD  research  and 
development  organization  in  comparison  to  ISO  9000.  Other  prevalent  systems  today, 
such  as  Six  Sigma  and  CMMI,  have  evolved  out  of  specific  disciplines,  and  therefore, 
retain  the  focus  of  that  area  as  their  underlying  process  methodology,  often  limiting  their 
utility  in  other  applications.  ISO  9000  underwent  an  analogous  evolution  from  the 
manufacturing  sector,  beginning  with  the  1994  version  and  evolving  to  the  2000  version. 
The  major  difference  of  ISO  9000  as  opposed  to  CMMI  or  Six  Sigma  is  that  it  is 
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supported  by  a  significant  external  standardization  development  body  and  has  the  benefit 
of  a  much  larger  audience  of  practitioners  providing  expanded  views  and  inputs  into  the 
standards  development  process. 

With  the  cancellation  of  the  MIL-Q-9858  and  MIL-I-45208  standards  in  1995,  the 
major  DoD-driven  quality  standards  were  eliminated  in  favor  of  commercial  standards. 
ISO  9001  was  judged  to  be  more  comprehensive  than  MIL-Q-9858  as  ISO  9002  (now 
obsolete  and  replaced  by  ISO  9001:2000)  was  deemed  a  suitable  replacement  for  MIL-I- 
45208. 74  Although  these  standards  were  probably  not  initially  written  for  internal 
government  consumption,  they  could  be  considered  as  alternatives  for  establishing  a 
quality  management  system. 

The  components  of  the  ISO  9001:2000  standard  have  been  generalized  to  the 
point  that  they  should  be  broadly  applicable  to  most  organizations,  providing  enough  of  a 
framework  to  ensure  a  complete  organizational  analysis  is  conducted,  but  also  being 
flexible  enough  to  accommodate  the  peculiarities  of  individual  organizations’  require¬ 
ments.  In  fact,  the  ISO  organization  refers  to  the  ISO  9000  and  ISO  14001  series  of 
standards  as  “generic  standards.”  “‘Generic’  also  signifies  that  no  matter  what  the 
organization's  scope  of  activity,  if  it  wants  to  establish  a  quality  management  system  or 
an  environmental  management  system,  then  such  a  system  has  a  number  of  essential 
features  for  which  the  relevant  standards  of  the  ISO  9000  or  ISO  14000  families  provide 

the  requirements. ”75 

The  aspect  of  international  recognition  of  the  ISO  9000  standards  also  provides  an 
added  benefit  as  DoD  enterprises  seek  to  leverage  foreign  government  investments  in 
defense  technology  to  offset  constrained  research  and  development  funding  for  mili¬ 
tary  system  procurement.  The  maturity  of  the  ISO  standard,  combined  with  a  robust 


74  Defense  Supply  Center  Columbus,  Defense  Logistics  Support  Command,  “ISO  9000  Quality  System 
Standards,  Application  in  DoD,”  http://www.dscc.dla.mil/library/qiiality  assurance/QAlibrary/ 

dod  and  dla  quality  issues/iso  disc  iso  9000  presentation.ppt.  Accessed  May  22,  2004. 

75  International  Organization  for  Standardization,  “ISO  9000  and  ISO  14000  -  in  brief,”  para  5, 
http://www.iso.ch/iso/en/iso9000-14000/index.html.  Updated  September  2003,  Accessed  May  22,  2004. 
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standards  development  and  maintenance  organization,  suggests  that  it  should  be  strongly 
considered  by  a  DoD  research  and  development  organization  seeking  to  implement  a 
quality  management  system. 

C.  ALTERNATE  QUALITY  MANAGEMENT  SYSTEMS 

A  wide  array  of  organizational  performance  improvement  tools  are  available 
today,  and  an  even  wider  array  of  consultants  are  available  to  help  organizational  leader¬ 
ship  in  their  implementation.  Examples  of  organizational  performance  improvement 
tools  include  Balanced  Scorecard,  Benchmarking,  Business  Process  Reengineering, 
Management  by  Objectives  (MBO),  Learning  Organization,  Knowledge  Management, 
Strategic  Planning,  Six  Sigma  and  Total  Quality  Management  (TQM). 76  Each  tool  has  a 
particular  focus  based  on  the  objectives  of  the  organization  in  the  context  of  its  current 
performance  and  the  changes  required  to  improve  perfonnance. 

The  duty  of  an  organization’s  senior  leadership  is  to  select  the  appropriate  tool  to 
employ  based  on  the  desired  changes  in  organizational  performance.  Tools  such  as 
Balanced  Scorecard  and  Benchmarking  emphasize  measurement  of  organizational 
performance  parameters,  but  typically  operate  in  a  reactive  fashion  since  they  only 
measure  past  performance.  Business  Process  Reengineering  aims  at  fundamentally 
changing  an  organization’s  processes  and  culture  to  achieve  significant  results  and  is 
usually  employed  in  a  context  of  a  significant  threat  to  organizational  survival. 

Quality  management  systems  should  be  considered  a  subset  of  organizational 
performance  improvement  tools  that  focus  on  the  quality  of  an  organization’s  products 
and  services  as  an  indicator  of  organizational  performance.  From  the  RSDBU 
perspective,  the  decision  to  implement  a  quality  management  system  was  primarily 
driven  by  the  desire  to  improve  organizational  financial  performance,  which  made  quality 
management,  and  specifically  ISO  9000,  a  requirement.  However,  there  was  also  a  desire 
to  standardize  and  document  the  processes  of  the  organization,  which  most  closely  aligns 
with  a  quality  management  approach  in  comparison  to  other  types  of  available  tools. 


76  Carter  McNamara,  MBA,  PhD,  “Broad  Overview  of  Various  Programs  and  Movements  to  Improve 
Organizational  Performance,”  Management  Assistant  Program  for  non-profits,  Copyright  1999, 
http://www.mapnp.org/library/org  perf/methods.htm.  Accessed  May  22,  2004. 
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Accordingly,  consideration  in  this  analysis  is  given  to  a  concurrent  project  man¬ 
agement  approach  that  was  the  previous  organizational  approach  to  performance 
improvement.  Also  considered  is  Total  Quality  Leadership  (TQL)  (the  Navy  imple¬ 
mentation  of  TQM)  at  the  strategic  level  based  on  the  Navy’s  broad  implementation  in 
the  1990s.  The  Capability  Maturity  Model  Integration  (CMMI)  was  also  considered 
during  the  RSDBU  process  development  initiative,  and  therefore,  warrants  attention. 
Finally,  the  Six  Sigma  approach,  which  is  an  alternative  commercial  approach  to  quality 
improvement,  merits  discussion  based  on  its  increasing  acceptance  in  aviation  logistics 
organizations  as  an  alternative  to  ISO  9001. 

To  evaluate  the  advantages  and  disadvantages  of  a  given  quality  management 
approach  for  an  organization,  it  is  necessary  to  first  examine  the  operating  environment 
of  the  organization  in  question  in  terms  of  policy  and  incentives.  In  the  case  of  the 
RSDBU,  there  is  no  current  policy  guidance  on  the  use  of  a  quality  management  system 
or  quality  standard  internal  to  the  government  functions  of  the  organization.  While  there 
is  substantial  guidance  in  the  FAR  and  DFARS  on  the  use  of  quality  management  in 
contracting,  it  does  not  apply  to  DoD  internal  operations.  As  further  evidence,  searches 
for  policy  directives  and  instructions  through  both  NAVSEA  and  NUWC  Division 
Newport  publications  resources  reveal  no  policy  guidance  specifically  related  to  quality 
management  applicable  to  government  components  of  the  organization. 

At  the  strategic  level,  the  last  significant  steps  taken  toward  formalizing  a  quality 
management  culture  within  the  government  began  in  the  first  administration  of  President 
Clinton  and  were  based  on  conclusions  of  the  National  Performance  Review  (NPR) 
program.  During  the  same  timeframe,  the  Department  of  Defense  undertook  its  own 
complementary  review  process,  the  Defense  Performance  Review  (DPR).  “The  DPR's 
objective  [was]  to  devise  innovative  ways  to  encourage  more  businesslike  practices  and 
market-driven  efficiencies  throughout  the  department.”77  One  of  the  twelve 


77  Government  Documents  Depart.  University  of  Northern  Texas,  “Accompanying  Report  of  the 
National  Performance  Review,  Department  of  Defense:  Part  1  of  2,”  30  September  1993, 
http://govinfo.librarv.unt.edLi/npr/librarv/nprrpt/annrpt/agnrpt93/dodl.html.  Accessed  May  22,  2004. 
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recommended  actions  from  the  DPR  report  issued  in  September  1993,  was  labeled 
“DOD08:  Establish  a  Defense  Quality  Workplace.”78 

The  Navy’s  implementation  of  this  DoD  quality  initiative  was  the  expansion  of 
the  TQL  program  from  logistics  elements  to  the  entire  service.  Begun  in  1984,  TQL  was 
based  on  “the  leadership  principles  of  Dr.  W.  Edwards  Deming  as  the  ‘leverage’  for  a 
quality  revolution  within  DOD”79  The  TQL  initiative  began  with  “a  number  of  actions  to 
ensure  that  ultimately  TQM/TQL  will  become  ‘the  normal  way  DOD  accomplishes  its 
mission. ’”80  The  Navy  established  a  Total  Quality  Leadership  Office  in  the  Washington, 
DC,  area  to  provide  products  and  services  for  consulting,  education  and  training,  infor¬ 
mation  and  communication,  assessment,  networking  and  liaison,  and  new  technologies.81 
The  TQL  office  was  intended  to  be  the  service- wide  resource  to  disseminate  TQL 
principles  for  organizations  to  apply  to  their  specific  functions. 

However,  the  Navy’s  TQL  efforts  never  seemed  to  gain  any  significant  traction 
and  were  dissolved  in  the  late  1990’s  with  no  defined  program  in  place  as  a  successor. 
While  broad  guidance  emphasizing  quality  remains  at  the  DoD  and  Navy  senior  leader¬ 
ship  levels,  there  is  no  specific  implementation  reaching  down  to  the  activity  level. 
Government  quality  imperatives  have  continued  to  lack  emphasis  in  the  George  W.  Bush 
administration  with  the  disestablishment  of  the  Presidential  Award  for  Quality  in  2002. 
Established  in  1988,  this  award  was  a  modified  version  of  the  Baldrige  Award,  the 
national  private-sector  quality  award  named  for  the  late  Malcolm  Baldrige,  who  served  as 
Secretary  of  Commerce  from  1981  to  1987. 82 

Therefore,  in  an  undefined  policy  environment,  the  assessment  of  alternatives  for 
a  quality  system  implementation  must  be  based  on  the  operating  environment  and 


78  Ibid. 

79  Linda  Doherty,  Director,  DON  Total  Quality,  “TQL1303  —  DoD  Formalizes  Commitment  To 
TQL/TQL,”  The  Department  of  The  Navy's  Total  Quality,  Leadership  Newsletter,  Issue  13,  March  1995, 
http://www.chinfo.navy.mil/navpalib/qualitv/tqleader/Tqleader.txt,  Accessed  May  22,  2004. 

80  Ibid. 

81  Joan  Kraft,  The  Department  of  the  Navy  Benchmarking  Handbook:  A  Systems  View,  Department  of 
the  Navy  Headquarters,  TQLO  Publication  Number  97-03,  28  May  1997,  P.2,  http://www.hq.navy.mil/ 
RBA/b mkhnbk.pdf.  Accessed  May  22,  2004. 

82  Stephen  Barr,  “Potent  Symbol  of  Quality  Management  Headed  for  Chopping  Block,”  Washington  Post, 
28  February  2001,  P.  B02,  http://www.washingtonpost.com.  Accessed  May  28,  2004. 
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incentives  of  the  individual  organization.  Beginning  with  the  RSDBU  example,  it  has 
been  established  that  policy  guidance  is  lacking  and  that  the  incentives  of  a  NWCF 
organization  place  an  emphasis  on  financial  perfonnance  of  the  organization. 

During  the  evaluation  phase  of  the  Code  70  process  documentation  initiative, 
other  alternatives  were  considered  as  an  augmentation  to  ISO  9001:2000,  but  the 
requirements  of  the  FMS  business  base  made  achieving  ISO  9001:2000  certification  a 
priority  in  order  to  pursue  international  business  opportunities. 

Along  with  a  lack  of  policy  guidance,  there  was  also  an  absence  of  historical 
reference.  Previous  organizational  initiatives  that  most  closely  aligned  with  a  quality 
management  system  had  been  focused  on  project  management  and  systems  engineering 
methodologies.  These  approaches  primarily  emphasized  optimizing  the  technical  design 
process,  as  opposed  to  documenting  and  improving  overall  organizational  processes  and 
their  impacts  on  quality. 

The  most  recent  example  of  such  an  approach  prior  to  the  RSDBU  ISO 
9001:2000  certification  initiative  was  based  on  the  work  of  University  of  Rhode  Island 
Professor  Quentin  C.  Turtle.  His  book,  titled  Implementing  Concurrent  Project 
Management,  is  intended  to  provide  a  “complete  procedure  for  implementing  concurrent 
engineering  in  the  planning,  scheduling  and  controlling  of  technical  projects.”83  Quality 
is  addressed  from  a  quality  assurance  perspective  and  emphasizes  planning  for  optimi¬ 
zation  of  quality  assurance  processes  in  the  form  of  more  rigor  in  estimating  quality 
assurance  requirements  during  the  planning  phase  and  a  commensurate  inspection 
protocol  during  design  and  production  phases.84 

The  major  difference  in  the  Turtle  approach  compared  to  ISO  9000  is  that  Turtle 
is  primarily  focused  at  the  product  level,  as  opposed  to  the  ISO  9000  perspective  of 
quality  at  an  organizational  level.  ISO  9000  uses  a  broader  emphasis  on  the  influence  of 
external  environmental  factors,  internal  work  processes,  and  measurement  of  customer 
satisfaction  as  indicators  of  quality  in  contrast  to  the  Turtle  approach  of  planning  for 

83  Quentin  Turtle,  “Implementing  Concurrent  Project  Management,”  PTR  Prentice  Hall,  Englewood 
Cliffs,  N.J.,  1994,  ISBN  0133020010,  P.  ix. 

84  Ibid.,  P.  8. 
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product  measurement  activities.  In  fact,  the  Turtle  approach  could  likely  be  used  as  a 
methodology  for  establishing  an  ISO  9000  compliant  design  control  process  if  its  tenets 
were  translated  into  the  work  instruction  package,  but  would  not  satisfy  the  full  spectrum 
of  functions  required  for  ISO  9000  certification. 

The  concurrent  project  management  model  emphasizes  integration  across  major 
business  functions,  including  marketing,  finance,  human  resources,  materials  manage¬ 
ment,  research  and  development,  quality  assurance,  and  manufacturing. 85  As  such,  its 
methodologies  are  constructed  and  sequenced  to  provide  optimal  value  in  a  commercial 
product  development  setting  where  time  to  market  and  lowest  cost  are  major  driving 
factors.  In  this  model,  the  time  to  market  target  is  set  at  one  year  from  design  to 
production.  Also,  concurrent  project  management  assumes  that  a  single  organization  has 
cognizance  over  the  entire  life  cycle,  from  design  and  development  through  production. 

It  is  the  authors’  experience  that  the  concurrent  project  management  approach  was 
generally  inconsistent  with  the  operations  of  the  RSDBU.  It  contains  several  useful 
methodologies  for  project  planning,  such  as  cost  estimation,  functional  specification 
development,  and  critical  path  definition.  However,  many  of  the  other  facets  of  this 
approach  are  geared  to  organizations  that  operate  in  a  defined  business  area  with  specific 
products  or  product  families  whose  requirements  are  driven  by  market  forces. 

Likewise,  the  assumption  in  the  Turtle  model  of  a  single  organization  having 
control  over  the  product  lifecycle  is  also  inconsistent  with  RSDBU  operations,  since  most 
RSDBU  products  are  single  unit  deliveries  whose  design  and  production  are  collaborative 
efforts  amongst  other  industry  partners  and  government  activities.  RSDBU  members  also 
have  limited  control  over  the  human  resources  and  finance  portions  of  the  Turtle  model, 
further  reducing  its  relevance. 

Finally,  the  RSDBU  business  base  does  not  correlate  well  to  the  concurrent 
project  management  approach  due  to  the  diversity  of  customer  requirements,  the 
extended  time  between  similar  projects  and  the  small  number  of  customers  for  RSDBU 
products.  This  results  in  generally  unique  products  that  have  a  long  lifespan  (20  years  for 


85  Ibid.,  P.  7. 
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some  components),  which  place  an  emphasis  on  quality  and  durability  above  the  schedule 
drivers  of  the  Turtle  model  in  comparison  to  commercial  products,  which  might  be 
expected  to  more  closely  follow  the  Turtle  model. 

Another  process  approach  that  was  considered  during  the  RSDBU  process  docu¬ 
mentation  initiative  was  the  use  of  the  CMMI  developed  by  the  Software  Engineering 
Institute  (SEI)  of  Carnegie  Mellon  University.  Although  CMMI  was  initially  conceived 
as  a  software  development  process  tool,  its  approach  is  extensible  to  other  product 
activities  since,  like  ISO  9000,  CMMI  is  focused  on  quality  and  process  management.  In 
fact,  a  version  of  CMMI  called  the  CMMI  Acquisition  Module  (CMMI-AM)  has  been 
developed  for  government  acquisition  organizations  to  “provide  a  foundation  for 
acquisition  process  discipline  and  rigor  that  enables  product  and  service  development  to 
be  repeatedly  executed  with  high  levels  of  ultimate  acquisition  success. ”86 

The  documented  version  of  the  CMMI-AM  process  is  relatively  new  in  com¬ 
parison  to  the  RSDBU  ISO  9000  certification  effort,  having  been  released  in  February 
2004,  three  years  after  the  RSDBU  certification  initiative.  However,  the  concept  of  using 
a  CMMI-based  approach  was  part  of  the  RSDBU  consideration  of  alternatives,  especially 
since  software  development  is  a  significant  portion  of  the  RSDBU  business  base.  The 
ISO  9001:2000  and  CMMI  models  have  been  cross-referenced  to  provide  a  form  of 
comparative  analysis  on  the  strengths  and  weaknesses  of  the  two  systems. 87 

The  contrasts  between  the  two  approaches  are  primarily  characterized  by  differing 
levels  of  focus  on  organizational  elements.  ISO  9000  is  more  focused  on  getting 
customer  feedback  and  measuring  satisfaction.  While  CMMI  doesn't  ignore  customer 
satisfaction,  it  is  not  strongly  emphasized  either.  ISO  9000  is  much  stronger  on  manage¬ 
ment  buy-in  and  participation  while  CMMI  is  primarily  focused  on  only  initial 
management  buy-in.  ISO  9000  calls  for  clear  internal  and  external  communications  plans 
while  CMMI  doesn't  really  address  this  from  a  management  perspective. 


86  Tom  Bernard  (USAF  ASC/EN),  Brian  Gallagher  (SEI),  Roger  Bate  (SEI),  and  Hal  Wilson  (Northrop 
Grumman),  “CMMI®  Acquisition  Module  (CMMI-AM),”  Version  1.0,  February  2004  [Technical  Report, 
CMU/SEI-2004-TR-00 1 ,  ESC-TR-2004-001]  P.  7,  http://www.sei.cmu.edu/mib/documents/ 
04.reports/pdf/04tr001.pdf,  Accessed  May  28,  2004. 

87  Boris  Mutafelija  and  Harvey  Stromberg,  July  2003. 
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Also,  ISO  9000  is  more  consistent  overall  relative  to  control  and  oversight  of 
purchasing  tasks.  CMMI  covers  purchasing  tasks,  but  is  considered  weaker  in  this  area. 
And  while  delivery  and  maintenance  aspects  of  quality  management  are  present  in  both 
ISO  9000  and  CMMI,  they  are  considered  stronger  in  the  ISO  approach.  Finally,  another 
key  difference  between  both  the  original  CMMI  and  new  CMMI-AM  and  ISO  9001:2000 
is  the  emphasis  on  process  documentation.  CMMI  and  CMMI-AM  do  not  provide  a 
defined  methodology  for  documentation  of  work  processes  and  instructions,  while  ISO 
mandates  process  documentation  as  the  most  significant  component  of  a  quality  man¬ 
agement  system.  CMMI-AM  is  also  focused  on  the  execution  of  specific  tasks  at  the 
project  level  as  opposed  to  the  broad  organizational  process  approach  of  ISO  9001:2000. 

There  is  one  additional  alternative  system  from  the  commercial  world  worth 
considering,  which  is  the  Six  Sigma  approach.  Six  Sigma  was  originally  developed  by 
Motorola  in  1979,  as  a  method  to  reduce  manufacturing  defects  and  the  associated 
impacts  on  customer  loyalty.  “Users  of  the  Six  Sigma  methodology  have  demonstrated 
that  the  surest  way  to  improve  performance  is  to  systematically  identify  the  causes  of 
waste,  lost  productivity,  and  customer  dissatisfaction,  and  then  adjust  internal  processes 
to  eliminate  them.”88  It  employs  five  steps  for  process  optimization:  define,  measure, 
analyze,  improve  and  control. 

Much  like  ISO  9000,  Six  Sigma  started  in  the  manufacturing  world  and  has  since 
expanded  beyond  those  boundaries.  It  has  been  applied  to  the  banking  industry, 89 
electrical  power  distribution  and  even  in  a  public  school  district.90  Also,  strikingly 
similar  to  ISO  9000  are  the  hordes  of  consultants  springing  up  to  provide  services  in 
implementing  Six  Sigma  in  most  any  organization.  Within  the  Navy,  Six  Sigma 


88  Ali  Kiran  and  Celal  Kaplan,  Contributors,  “Six  Sigma  —  it's  not  just  for  manufacturers  anymore,” 
29  March  2004,  http://searchcio.techtarget.com/originalContent/0,289142,sidl9  gci9571 54,00. html. 
Accessed  May  28,  2004. 

89  Ibid. 

90  Mary  Ruff,  Six  Sigma,  “Using  Six  Sigma  to  Solve  Issues  in  Public  School  System,”  17  May  2004, 
http://www.isixsigma.com/librarv/content/c040517a.asp.  Accessed  May  30,  2004. 
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principles  are  being  applied  to  naval  aviation  depots  to  reduce  cycle  time  and  improve 
efficiency, 9i  including  those  same  depots  that  have  been  certified  to  ISO  9001. 

While  Six  Sigma  provides  a  valuable  framework  to  analyze  processes,  it  is  best 
applied  to  the  improvement  of  repetitive  processes  that  consistently  perfonn  the  same 
activities  and  have  large  quantities  of  output  products.  In  one  user’s  opinion:  “[i]t’s  best 
for  organizations  that  produce  lots  and  lots  of  small  parts  and  want  to  get  a  99.9%  quality 
rate  on  the  parts. ”92  The  examples  cited  above,  such  as  naval  aviation  depots,  banks,  and 
public  schools,  all  share  the  common  trait  of  producing  large  volumes  of  output  products 
with  a  desired  high  degree  of  conformity  in  the  results.  From  the  RSDBU  perspective  of 
mostly  one-off  system  production,  the  Six  Sigma  approach  would  not  likely  be  an 
effective  alternative  because  of  the  difficulties  of  developing  appropriate  statistical 
methods  applicable  to  a  limited  sample  of  diverse  products.  Furthermore,  because  the 
RSDBU  completes  mostly  one-off  system  production,  a  business  case  analysis  cannot  be 
made  for  Six  Sigma;  as  a  result,  Six  Sigma  is  inappropriate  for  the  RSDBU  operating 
environment.  As  Mikel  Harry  and  Richard  Schroeder  state  in  the  third  line  of  their  book 
on  Six  Sigma  process  and  application,  “Six  Sigma  is  about  improving  profitability  [or,  in 
the  case  of  the  government,  cost  reduction]93 

As  revealed  in  the  preceding  examples,  the  selection  of  a  quality  management 
system  should  be  based  on  the  business  base  and  operating  methodologies  of  the  organi¬ 
zation.  In  fact,  it  may  be  more  beneficial  to  consider  an  approach  that  supports  business 
process  optimization  during  the  initial  implementation  of  a  quality  management  system, 
since  the  numerous  systems  available  span  many  differing  aspects  of  business  process 
analysis.  However,  the  key  elements  that  can  be  seen  throughout  all  of  the  preceding 
models  are  the  need  for  organizational  introspection  and  environmental  analysis.  Some, 
such  as  Six  Sigma,  tend  to  be  more  focused  on  internal  process  analysis,  but  the 
boundaries  for  that  analysis  and  the  measures  of  success  will  be  determined  by  the 

91  Capt.  Tim  Trainer,  “NAVAIR  Airspeed  Brief  to  Dod  Maintenance  Symposium  &  Exhibition,”  Brief 
5  September  2003,  http://www.sae.org/events/dod/presentations/2003capttimtrainer.pdf.  Accessed  May  28, 
2004. 

92  Richard  Lutman,  McLaughlin  Research  Corporation,  e-mail,  dated  17  May  2004. 

93  Harry  Mikel  J.  and  Richard  Schroeder,  Six  Sigma:  The  Breakthrough  Management  Strategy 
Revolutionizing  the  World's  Top  Corporations,  Doubleday,  2000. 
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external  organizational  environment.  All  of  the  models  previously  discussed  essentially 
seek  to  drive  an  organization  to  align  its  internal  processes  with  the  external  operating 
environment  based  on  the  policy  and  incentives  of  the  organization. 

There  is  no  “one  size  fits  all”  solution  for  improving  the  perfonnance  of  an 
organization;  each  system  has  advantages  and  disadvantages  that  must  be  weighed 
against  organizational  objectives.  It  is  incumbent  on  senior  management  to  identify  the 
external  imperatives  of  the  organization’s  operating  environment  and  select  the  appro¬ 
priate  tool  to  correlate  the  organization’s  outputs  with  those  imperatives.  External 
imperatives  may  be  defined  as  quantity,  cost,  schedule,  time  to  market,  defect  rate,  or  any 
number  of  other  parameters  or  combinations  of  those  parameters.  An  organizational 
performance  improvement  tool,  such  as  a  quality  management  system,  should  initially 
facilitate  business  process  optimization  and  enable  continuing  improvement  of  those 
processes  based  on  the  priorities  of  customers  and  stakeholders.  Furthennore,  the  system 
should  be  flexible  enough  to  be  tailored  in  such  a  way  as  to  identify  the  tradeoffs  amongst 
all  facets  of  an  organization’s  operations  to  maximize  success. 

For  the  senior  management  of  the  Engineering  Test  and  Evaluation  Department, 
ISO  9001:2000  represented  the  best  choice  for  a  quality  management  system  for  the 
RSDBU  based  on  the  applicability  of  the  ISO  9001:2000  approach  to  the  RSDBU 
business  base  and  organizational  imperatives.  ISO  9001:2000  provided  the  flexibility  to 
address  the  diversity  of  products  developed  by  the  RSDBU  and  enabled  the  integration  of 
existing  regulatory  and  policy  requirements  into  the  processes  developed  to  achieve 
certification.  Priorities  of  customers  and  stakeholders  were  accommodated  in  a  manner 
that  did  not  introduce  a  significant  additional  burden  on  teams  executing  programs.  This 
allowed  internal  organizational  priorities  of  management  oversight  and  the  support  of  the 
RSDBU  core  equity  of  undersea  range  development  to  be  balanced  against  the  external 
imperatives  of  efficiency  and  quality. 

D.  ISO  AND  STRUCTURED  AND  REGULATED  ORGANIZATIONS 

The  generic  approach  of  the  ISO  9001  standard  supports  its  use  in  an  organization 
that  is  highly  structured  and  regulated  by  accounting  for  existing  regulatory  and  policy 

guidance  as  part  of  an  organization’s  processes.  Since  one  of  the  principal  tenets  of  ISO 
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is  process  documentation,  the  established  and  well  documented  structure  of  the  DoD 
acquisition  process  satisfies  many  of  the  ISO  requirements.  This  allows  a  DoD  organiza¬ 
tion  pursuing  certification  to  reference  existing  policy  and  regulations  as  process 
documentation  without  having  to  devote  time  and  resources  to  the  development  of 
additional  and  unnecessary  procedures. 

While  this  aspect  of  ISO  9001  may  facilitate  certification,  attempting  to  develop  a 
common  quality  system  for  a  large  organization  operating  in  a  highly  regulated  envi¬ 
ronment  would  likely  prove  to  be  prohibitively  complex  and  expensive.  From  the 
regulatory  and  policy  perspective  within  DoD,  there  is  the  DFARS;  annually  changing 
congressional  language;  acquisition  policy  guidance  of  the  DoD  5000  series  of  instruc¬ 
tions;  Chairman,  Joint  Chiefs  of  Staff  instructions;  and  additional  service-specific  policy 
guidance  and  procedures.  Attempting  to  correlate  all  of  these  inputs  into  a  coherent  set  of 
ISO  9001:2000  process  documentation  would  be  a  substantial  task  that  has  the  potential 
to  add  yet  another  collection  of  documentation  to  an  already  saturated  environment 
before  even  considering  the  significant  expense  of  achieving  and  maintaining 
certification  at  a  department  or  agency  level. 

Indeed,  the  argument  has  been  made  that  ISO  9000  should  be  internalized  in 
government  service: 

It's  not  that  those  in  government  aren't  aware  of  ISO  9000.  They  like  it 
well  enough  to  impose  quality  management  systems  based  on  ISO  9000 
on  their  contractors.  Regulatory  agencies  also  think  ISO  9000  is  good 
enough  to  promote  in  their  regulated  industries...  If  those  in  government 
recognize  that  ISO  9000  is  good  for  industry,  why  don't  they  believe  it's 
good  for  government?  Clearly,  they  need  to  rethink  ISO  9000’s  possible 
applications  in  government  and  consider  implementing  it  as  their  internal 
quality  management  system.  ISO  is  as  applicable  to  service  providing 
organizations  as  it  is  to  manufacturing  organizations. 94 

While  this  argument  has  prima  facie  merit,  it  fails  to  consider  the  complexities  of 
ISO  9000  implementation  that  have  been  demonstrated  in  the  business  world.  The  ISO 
9000  implementation  for  Polyguard  Products,  a  company  with  two  dozen  employees  and 


94  Ira  Epstein,  “Raising  Federal  Standards,”  Quality  Digest,  July  1999,  http://www.qualitydigest.com/ 
iulv99/html/bodv  epstein.html.  Accessed  May  28,  2004. 

79 


$5M  in  annual  sales,  took  $30,000  and  one  year  in  1996.95  Extrapolating  those  resource 
requirements  to  an  agency  the  size  of  DoD  leads  to  a  financial  obligation  and  schedule 
requirement  that  would  invite  unwanted  congressional  and  media  attention  to  an 
investment  with  an  unquantifiable  return. 

Likewise,  there  is  also  an  issue  of  public  perception  that  all  government  activities 
must  face.  By  virtue  of  being  a  taxpayer- funded  organization,  one  of  the  prime  external 
imperatives  of  government  service  is  the  demonstration  of  being  a  responsible  steward  of 
the  taxpayer’s  money.  For  an  organization  such  as  NASA,  their  greater  external  impera¬ 
tive  is  the  maintenance  of  America  at  the  forefront  of  space  travel  and  exploration,  so  the 
quality  of  their  products  in  terms  of  safety  and  capability  places  financial  accountability 
as  a  secondary  external  imperative.  So  from  the  NASA  perspective,  an  organization- 
wide  implementation  of  ISO  9001  could  realistically  be  perceived  and  portrayed  as  a 
worthwhile  investment  in  demonstrating  a  commitment  to  their  foremost  organizational 
imperative. 

For  DoD,  a  major  criticism  often  received  is  a  lack  of  efficiency.  Former  Vice 
President  Gore’s  Hammer  Award  was  a  well-publicized  symbol  of  inefficiencies  and 
waste  attributed  to  DoD  procurement  through  the  purchase  of  $435  hammers.96 
Therefore,  a  driving  external  imperative  for  DoD  is  efficiency  in  defense  procurement, 
with  quality  being  a  close  second.  And  since  DOD  is  funded  by  Congress,  whose 
primary  function  is  financial  oversight,  DoD  is  most  often  reviewed  in  a  financial 
perspective. 

Based  on  the  conclusion  of  Chapter  3  that  a  cost-benefit  analysis  for  ISO  9000 
implementation  cannot  be  realistically  done  in  either  a  predictive  or  comparative  fashion 
for  a  government  organization,  it  is  unlikely  that  large  organizational  segments  of  DoD 
will  reflexively  embrace  ISO  9000  given  the  primacy  of  the  fiscal  efficiency  imperative. 
The  implementation  of  a  quality  management  system  within  this  environment  could  be 


95  Margaret  Allen,  15  August  1997. 

96  James  Fairhall,  “The  Case  for  the  $435  Hammer,”  University  of  Mississippi, 
http://home.olemiss.edu/~imitchel/class/hammer.htm,  [originally  appearing  in  The  Washington  Monthly, 
January  1987],  Accessed  May  28,  2004. 
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perceived  as  yet  another  expenditure  of  taxpayer  money  without  a  quantifiable  expecta¬ 
tion  of  benefits,  particularly  at  the  broad  level  of  the  DoD  mission.  Attempting  to  launch 
a  department-wide  certification  effort  without  an  unequivocally  quantifiable  return  on 
investment  is  unadvisable  given  the  operating  environment  of  DoD. 

This  assertion  is  borne  out  by  the  Navy’s  experience  with  TQL.  Although  the 
program  began  in  1984  as  a  performance  improvement  tool  for  Navy  logistics  organiza¬ 
tions, 97  the  attempt  to  deploy  TQL  across  the  Department  of  the  Navy  proved  to  be  a 
substantial  undertaking  that  met  with  limited  success.  From  the  logistics  standpoint, 
naval  aviation  depots  have  moved  on  to  ISO  9000  certification  and  are  exploring  the  use 
of  Six  Sigma  as  their  implementations  of  quality  management.  The  quiet  evaporation  of 
the  Navy  TQL  program  in  the  late  1990s,  leaving  little  behind  as  an  organizational 
legacy,  bears  witness  to  the  complexities  of  implementing  a  quality  system  at  such  a  large 
scale. 

The  foregoing  analysis  and  examples  should  not  be  viewed  as  a  wholesale  indict¬ 
ment  of  the  use  of  ISO  9001:2000,  or  other  appropriate  quality  systems,  within  a  highly 
structured  and  regulated  organization  such  as  DoD.  In  the  authors’  opinion,  ISO 
9001:2000  has  been  an  effective  tool  for  the  RSDBU  for  the  documentation  and 
improvement  of  internal  business  processes  to  make  the  RSDBU  organization  more 
efficient  and  responsive  to  sponsors  and  stakeholders.  However,  the  level  of  the  organi¬ 
zation  at  which  ISO  9001:2000  is  targeted  should  be  selected  judiciously  based  on 
common  organizational  functions  or  business  areas  using  a  definable  level  of  com¬ 
monality  that  ensures  the  relevance  of  the  quality  management  system  to  the  employees 
who  will  be  using  it. 

Therefore,  the  definable  level  of  commonality  must  be  established  on  a  case-by- 
case  basis  for  each  organizational  component  considering  the  implementation  of  ISO 
9001:2000.  For  the  Engineering,  Test  and  Evaluation  Department,  the  parent  organiza¬ 
tion  of  the  RSDBU,  the  business  units,  of  which  the  RSDBU  is  one,  represented  the  best 


97  Archester  Houston,  Ph.D.  and  Steven  L.  Dockstader,  Ph.D.,  “Total  Quality  Leadership:  A  Primer,” 
Department  of  the  Navy,  Total  Quality  Leadership  Office,  TQLO  Publication  Number  97-02,  P.5, 
http://www.balancedscorecard.org/files/primer.pdf.  Accessed  May  28,  2004. 
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level  of  organizational  commonality  for  the  development  of  an  ISO  9001:2000  quality 
management  system.  This  is  driven  by  the  fact  the  Engineering,  Test  and  Evaluation 
Department  is  a  fairly  heterogeneous  organization  composed  of  many  differing  technical 
and  engineering  specialties  serving  numerous  discrete  business  bases  of  NUWC  Division 
Newport.  For  example,  the  Technology  Applications  and  Development  Division  is 
composed  of  employees  operating  in  three  diverse  specialties:  undersea  range  systems 
development  (supporting  the  RSDBU),  reliability  and  system  safety  analysis,  and 
physics-based  modeling.  It  is  unlikely  that  a  highly  effective  common  quality  system 
could  be  developed  that  would  be  relevant  to  the  organizational  missions  of  these  three 
groups. 

The  structure  and  regulatory  enviromnent  of  the  RSDBU  served  to  simplify  the 
certification  process  by  referencing  existing  documentation,  where  appropriate,  to  fulfill 
ISO  9001:2000  requirements.  This  efficiency  can  be  replicated  by  other  DoD  organiza¬ 
tions  that  wish  to  pursue  certification  as  well.  However,  the  overall  effectiveness  of  the 
quality  management  system  will  be  determined  by  the  applicability  of  the  developed 
processes  to  the  organization’s  mission  and  operating  environment.  This  requires  that  the 
scope  of  the  organizational  segments  to  be  certified  be  carefully  selected  to  maximize  the 
relevance  of  the  quality  management  system  to  employees’  duties  and  thereby  maximize 
the  organizational  improvements  achieved. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  INTRODUCTION 

The  ISO  9000  quality  management  standards  are  internationally  recognized  as 
tools  for  the  improvement  of  organizational  performance.  Within  the  United  States 
Department  of  Defense  acquisition  community,  the  benefits  of  quality  management 
standards  have  been  recognized  since  the  establishment  of  the  MIL-Q-9858  quality 
management  program  in  1959.  While  quality  management  standards  and  programs  have 
long  been  applied  to  contractors  supplying  DoD,  there  is  no  definitive  guidance  on  the 
use  of  quality  management  standards  within  DoD. 

The  Defense  Department  maintains  a  large  collection  of  research  and  develop¬ 
ment  organizations  that  specialize  in  fields  and  disciplines  critical  to  national  defense,  as 
well  as  organizations  with  numerous  other  functions.  Those  organizations  primarily 
specializing  in  research  and  development  focus  on  the  specialized  needs  of  the  military 
where  there  is  insufficient  capacity  in  industry  or  the  technologies  are  too  specialized  or 
sensitive  for  application  to  the  commercial  sector.  Ultimately,  the  output  of  these  organi¬ 
zations  is  translated  into  equipment,  systems  and  services  that  are  provided  to  the  military 
to  enable  the  execution  of  their  assigned  missions. 

The  application  of  an  ISO  9000  quality  management  system  within  one  such  DoD 
activity,  the  Range  Systems  Development  Business  Unit  (RSDBU)  of  the  Engineering, 
Test  and  Evaluation  Department  of  the  Naval  Undersea  Warfare  Center  Division 
Newport  has  shown  that  the  ISO  9001:2000  standard  can  be  an  effective  organizational 
performance  improvement  tool  for  a  DoD  research  and  development  activity.  However, 
ISO  9001:2000  certification  should  not  be  considered  a  panacea  for  organizational 
performance  or  output  problems;  the  attainment  of  certification  is  primarily  a  symbolic 
achievement  that  must  have  long-tenn  management  commitment  to  be  a  truly  effective 
component  of  organizational  perfonnance  improvement. 

The  RSDBU  organizational  imperatives  of  quality  and  cost-effectiveness  are 
derived  from  the  incentives  of  the  NWCF  organizational  structure  under  which  NUWC 
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Division  Newport  operates  and  the  peculiarities  of  the  business  base  for  undersea  range 
technology  development.  The  NWCF  is  a  revolving  fund  that  relies  on  funded  tasking 
from  external  customers  rather  than  direct  appropriations  to  finance  its  operations.  As 
such,  NUWC,  and  therefore  the  RSDBU,  must  generate  sufficient  revenue  to  cover  the 
full  costs  of  operations  and  finance  continuing  operations.  Within  these  constraints, 
NUWC  must  operate  on  a  break-even  basis  over  time  by  neither  making  a  profit  nor 
incurring  a  loss.  The  business  base  of  undersea  tracking  range  system  development  has  a 
customer  base  is  limited  to  DoD  and  is  characterized  by  diverse  customers  and  require¬ 
ments  for  highly  specialized  systems  with  highly  aperiodic  modernization,  repair  and 
replacement  schedules. 

B.  CONCLUSIONS 

ISO  9001:2000  represents  one  alternative  in  a  broad  collection  of  organizational 
performance  improvement  tools  focused  on  quality.  When  choosing  an  organizational 
performance  tool,  the  selection  must  be  based  on  an  organization’s  operating  environ¬ 
ment  and  incentives  base  to  ensure  that  the  implementation  investment  provides  maxi¬ 
mized  benefit.  For  the  RSDBU,  the  primary  incentive  base  is  the  retention  and  expansion 
of  the  customer  base  by  delivering  quality  products  and  services  in  a  financially  efficient 
manner. 

The  operating  environment  for  the  RSDBU  continues  to  evolve  in  a  manner  that 
not  only  justifies  the  initial  investment  to  achieve  certification,  but  warrants  continued 
investment  in  the  maintenance  and  improvement  of  the  RSDBU  quality  management 
system.  Within  this  environment,  the  RSDBU  implementation  of  ISO  9001:2000  has 
proven  to  be  a  sound  investment  in  the  establishment  of  a  quality  management  system 
that  helps  to  facilitate  the  achievement  of  organizational  objectives  in  a  cost  effective 
manner.  The  original  motivations  for  establishing  documented  processes  were  largely 
realized,  and  the  operation  and  improvement  of  the  system  over  the  succeeding  three 
years  has  demonstrated  management  commitment  to  the  retention  of  ISO  9001:2000 
certification. 

As  seen  in  the  RSDBU  program  examples  cited  in  Chapter  II,  the  principal  benefit 

of  an  ISO  9001:2000  quality  management  system  has  been  the  establishment  of  a 
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common  set  of  management  oversight  processes  and  practices.  As  a  research  and 
development  organization  that  operates  across  a  relatively  broad  span  of  disciplines  and 
customers,  the  application  of  a  common  program  execution  process  based  on  ISO  9000 
has  established  a  consistent  methodology  for  program  performance  monitoring  which 
serves  to  set  a  baseline  of  expectation  management  for  both  employees  and  management. 

While  substantial  tailoring  for  specific  RSDBU  program  requirements  is  often 
necessary,  the  generalized  approach  of  ISO  9001:2000  provides  a  more  flexible  frame¬ 
work  than  has  been  used  in  the  past.  Previous  management  monitoring  approaches  have 
generally  been  the  imposition  of  mandated  commercial  program  management  tools  that 
focus  only  on  the  technical  design  aspect  of  programs.  However,  these  specialized 
systems  tend  to  become  less  applicable,  and,  therefore,  less  efficiently  useable  over  time 
as  technical,  regulatory  and  policy  environments  continue  to  evolve. 

The  RSDBU  ISO  9001:2000  has  demonstrated  its  utility  in  the  execution  of 
RSDBU  programs  and  also  revealed  some  areas  for  which  it  is  not  optimized,  such  as  the 
execution  of  a  development  program  in  a  highly  dynamic  external  environment.  The 
RSDBU  system  has  also  survived  changes  in  senior  management  leadership  since  initial 
certification  was  achieved.  The  Code  70  Department  Head  position  has  been  occupied  by 
two  successors  who  have  remained  committed  to  the  Code  70  ISO  quality  management 
system.  Changes  have  also  been  made  in  the  Code  70  ISO  Steering  Committee  member¬ 
ship,  other  senior  leadership  positions  and  the  RSDBU  team  leader  and  member 
composition  that  have  not  significantly  perturbed  the  overall  execution  of  the  process. 

In  part,  the  effectiveness  of  the  RSDBU  ISO  9001:2000  quality  management 
system  can  be  measured  by  surviving  organizational  evolution  over  time.  Although  three 
years  are  not  necessarily  a  substantial  period  in  some  measures,  the  drive  toward  trans¬ 
formation  in  the  national  defense  arena  that  has  percolated  down  to  the  lowest  levels  of 
most,  if  not  all,  DoD  organizations,  has  caused  significant  changes  at  NUWC  Division 
Newport  as  the  parent  organization  of  the  RSDBU. 

Changes  in  environment,  organization  and  management  generally  result  in  corre¬ 
sponding  changes  in  business  practices,  sometimes  with  small  realized  gains.  Perhaps 
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one  of  the  additional  side  benefits  of  the  ISO  9001  structure  of  third  party  registration  and 
periodic  accreditation  is  the  level  of  visibility  that  would  be  associated  with  the  loss  of 
certification  if  that  were  to  happen.  As  long  as  process  standardization  remains  an 
imperative  of  the  organization  as  a  whole,  an  incentive  is  created  to  maintain  an  existing 
certification.  From  a  working-level  productivity  perspective,  once  a  process  has 
stabilized  to  become  understood  and  accepted,  it  is  in  the  best  interest  of  the  organization 
to  retain  that  process  as  long  as  it  continues  to  provide  some  level  of  beneficial  output. 

The  path  to  ISO  9000  certification  of  the  RSDBU  quality  management  system 
was  relatively  straightforward,  although  this  is  not  the  case  for  all  organizations.  The 
RSDBU  implementation  of  ISO  9001:2000  has  proven  beneficial  to  the  organization  by 
providing  a  measurement  and  management  tool  that  has  been  tailored  to  the  overall  focus 
of  the  RSDBU  business  base.  By  definition,  the  system  cannot  be  optimized  for  every 
program  and  customer  due  to  the  diversity  of  the  business  base  that  drives  the  necessary 
generality  of  the  quality  management  system.  However,  the  initial  investment  in  a 
quality  management  system  can  often  be  daunting  for  an  organization  depending  on  the 
scope  of  the  processes  involved,  especially  since  it  is  difficult  to  specifically  quantify  the 
actual  business  performance  gains  achievable  from  the  system. 

The  impact  of  a  documented  quality  management  system  on  the  organizational 
output  of  a  government  enterprise  will  be  difficult  to  reliably  predict  prior  to  making  the 
investment  due  to  the  limitations  of  statistical  process  measurement  for  government 
functions.  Accurately  measuring  the  impact  of  a  quality  system  on  a  government  organi¬ 
zation’s  perfonnance  metrics  can  be  equally  difficult  once  it  has  been  established  for  this 
same  reason.  Gross  measures  may  be  derived  based  on  “before  and  after”  comparisons, 
but  attributing  detailed  numerical  quantities  such  as  a  reduction  in  operational  costs  or  a 
decrease  in  cycle  time  to  the  predicted  performance  of  a  quality  system  would  have  a 
highly  subjective  interpretation. 

ISO  9000  offers  a  defined  methodology  of  process  documentation  that  can  be 
used  for  organizational  performance  improvement  if  applied  correctly.  However,  ISO 
9000  may  initially  be  an  expensive  undertaking  because  of  the  scope  of  the  effort 

involved  in  process  documentation,  and  may  be  difficult  to  justify  financially  from  a 
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return-on-investment  perspective,  unless  a  rigorous  cost  analysis  system  is  already  in 
place  and  functioning  efficiently.  If  a  cost-benefit  justification  is  necessary,  the  business 
of  the  organization  should  also  be  such  that  “before  and  after”  comparisons  are  realistic 
and  based  on  consistently  measurable  quantities.  Cost  benefit  analyses  should  also 
reflect  that  once  the  initial  certification  is  achieved,  resources  must  be  devoted  to  main¬ 
taining  and  improving  the  quality  system. 

Therefore,  ISO  9000  may  not  be  a  significantly  better  management  tool  than 
many  of  the  other  quality  management  schemas  that  are  widely  available,  but  ISO  9000 
comes  with  the  added  benefit  of  international  recognition  that  allows  an  organization  to 
maximize  its  customer  base.  If  a  management  team  is  truly  motivated  to  improve  organ¬ 
izational  performance,  ISO  9000  can  be  a  valuable  tool  to  assess  the  complete  process. 
However,  the  final  decision  on  a  quality  management  system  should  be  based  on  the 
compatibility  of  the  tool  to  the  organization’s  goals  and  operating  methodologies, 
combined  with  the  demands  of  the  market  in  which  the  organization  operates.  If  “market 
coercion”  is  a  factor,  that  should  be  understood,  but  ISO  9000  will  provide  the  most 
benefit  if  it  is  used  as  a  tool  to  improve  organizational  performance,  as  opposed  to  a 
minimalist  approach  intended  primarily  to  mollify  customers. 

As  previously  discussed,  the  primary  financial  incentive  for  achieving  ISO  9001 
certification  for  the  RSDBU  was  to  enable  pursuit  of  FMS  work.  However,  since  the 
RSDBU  is  not  currently  engaging  in  any  FMS  work,  it  cannot  be  demonstrated  that 
funding  has  increased  through  this  avenue.  Similarly,  none  of  the  current  RSDBU 
customers  have  formally  made  ISO  9000  certification  a  precondition  for  providing  new 
tasking  or  preserving  existing  tasking,  so  it  also  cannot  be  shown  that  ISO  9000  has 
directly  contributed  to  the  expansion  or  preservation  of  the  RSDBU  business  base. 
Therefore,  the  merits  of  the  RSDBU  quality  management  system  must  be  determined  as  a 
subjective  assessment  as  to  whether  ISO  9001:2000  provides  benefit  through  improve¬ 
ment  of  internal  processes  that  maximize  value  to  the  customer. 

For  the  RSDBU,  ISO  9001:2000  represented  the  best  choice  for  a  quality  man¬ 
agement  system  based  on  the  applicability  of  the  ISO  9001:2000  approach  to  the  RSDBU 

business  base  and  organizational  imperatives.  ISO  9001:2000  provided  the  flexibility  to 
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address  the  diversity  of  products  developed  by  the  RSDBU  and  enabled  the  integration  of 
existing  regulatory  and  policy  requirements  into  the  processes  developed  to  achieve 
certification.  Priorities  of  customers  and  stakeholders  were  accommodated  in  a  manner 
that  did  not  introduce  a  significant  additional  burden  on  teams  executing  programs.  This 
allowed  internal  organizational  priorities  of  management  oversight  and  the  support  of  the 
RSDBU  core  equity  of  undersea  range  development  to  be  balanced  against  the  external 
imperatives  of  efficiency  and  quality. 

The  aspect  of  international  recognition  of  the  ISO  9000  standards  also  provides  an 
added  benefit  as  DoD  enterprises  seek  to  leverage  foreign  government  investments  in 
defense  technology  to  offset  reduced  research  and  development  funding  for  military 
system  procurement.  The  maturity  of  the  ISO  standard  combined  with  a  more  robust 
standards  development  and  maintenance  organization  supporting  its  evolution  suggests 
that  it  is  currently  the  best  of  the  available  alternatives  for  a  DoD  research  and  develop¬ 
ment  organization. 

The  structure  and  regulatory  enviromnent  of  the  RSDBU  served  to  simplify  the 
certification  process  by  referencing  existing  documentation  where  appropriate  to  fulfill 
ISO  9001:2000  requirements.  This  efficiency  can  be  replicated  by  other  DoD  organiza¬ 
tions  that  wish  to  pursue  certification  as  well.  However,  the  overall  effectiveness  of  the 
quality  management  system  will  be  determined  by  the  applicability  of  the  developed 
processes  to  the  organization’s  mission  and  operating  environment.  This  requires  that  the 
scope  of  the  organizational  segments  selected  for  certification  be  carefully  analyzed  to 
maximize  the  relevance  of  the  quality  management  system  to  employees’  duties;  thereby, 
maximizing  the  organizational  improvements  achieved. 

C.  RECOMMENDATIONS 

Following  are  the  authors’  recommendations  derived  from  the  results  of  this  case 

study: 

•  Government  organizations  seeking  ISO  9001 :2000  certification  should 

carefully  quantify  expectations  for  organization  performance  improvement 
with  the  understanding  that  benefits  of  certification  cannot  be  strictly 
quantified  in  financial  tenns. 
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•  All  organizations,  including  government  activities,  choosing  to  pursue  ISO 
9001:2000  certification  should  use  the  initial  process  documentation 
effort,  undertaken  as  part  of  the  development  of  an  ISO  9001:2000 
compliant  quality  management  system,  as  an  opportunity  to  analyze  and 
optimize  business  processes,  as  opposed  to  strictly  documenting  the 
existing  way  of  doing  business. 

•  Once  an  organization  has  adopted  an  ISO  9001 :2000  quality  management 
system,  the  continuous  improvement  process  should  include  an  assessment 
of  the  relevance  of  the  ISO  9000  approach  to  the  organization’s  existing 
and  projected  business  base. 

•  For  the  RSDBU  implementation,  the  minimum  elapsed  time  between 
program  reviews  should  be  six  months  as  opposed  to  the  current  one  year. 
This  will  provide  the  opportunity  for  management  to  initiate  any  necessary 
corrective  actions  within  the  annual  budget  cycle. 

•  Third  party  certification  of  government  ISO  9001 :2000  quality 
management  systems  is  important  and  should  be  required.  Although  DoD 
does  not  require  third-party  certification  of  contractors’  quality  systems, 
the  authors  believe  that  this  is  an  important  aspect  to  ensure  that 
organizational  management  is  motivated  to  maintain  the  system  and  retain 
its  relevance  to  workforce  activities. 

•  Implementation  of  an  ISO  9001 :2000  quality  system  within  a  DoD 
organization  should  be  tailored  to  the  organization’s  operating 
environment.  Federal  and  DoD  statutes,  regulations  and  policy  should  be 
complementary  to  an  ISO  9001:2000  quality  system  as  opposed  to  adding 
another  layer  of  bureaucracy. 

•  A  DoD  implementation  of  an  ISO  9001 :2000  quality  system  must  be 
targeted  at  a  manageable  level  for  the  organization  to  remain  relevant. 

The  greater  the  diversity  of  missions  within  the  organizational  unit  to  be 
certified,  the  greater  the  generalization  of  the  QMS,  which  thereby  reduces 
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the  applicability  to  individual  employees’  and  group’s  day-to-day 
functions. 

•  Navy  Working  Capital  Fund  activities  that  achieve  ISO  9001 :2000 
certification,  such  as  the  RSDBU,  should  demonstrate  the  value  of  an  ISO 
9001:2000  QMS  to  customers  and  stakeholders. 

•  An  ISO  900 1 :2000  quality  management  system  should  be  used  as  part  of  a 
broader  strategic  management  framework.  An  ISO  9000  QMS  can  fulfill 
the  needs  for  an  organizational  performance  evaluation  methodology,  but 
does  not  provide  all  of  the  necessary  components  for  effective  strategic 
planning. 

D.  RECOMMENDATIONS  FOR  FURTHER  STUDY 

An  area  for  further  study  includes  a  detailed  analysis  of  the  application  of  ISO 
9001  in  larger  organizations,  such  as  NASA.  A  key  focus  should  be  the  assessment  of 
the  point  at  which  the  effectiveness  of  ISO  9001  becomes  reduced  due  to  the  broad  scope 
of  organizational  processes  encompassed  by  a  single  system.  Corollary  analyses  could 
address  the  effectiveness  of  ISO  9001  in  both  commercial  and  government  applications 
as  a  function  of  population  of  the  certification  community  as  well  as  the  relative 
homogeneity  or  diversity  of  skill  base  qualifications  found  in  those  communities.  Ideally, 
it  should  be  possible  to  develop  general  guidance  for  an  organization  to  optimize  the 
implementation  and  operation  of  an  ISO  9001  system  using  a  set  of  metrics  such  as 
mission  scope,  population  and  skill  mix  to  identify  business  and  organizational  segments 
that  will  recognize  the  greatest  gains  from  a  certified  quality  management  system. 
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